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MEMORANDUM  FOR  NAVAL  INSPECTOR  GENERAL 

SUBJECT:  Report  on  Acquisition  of  the  EA-6B  Improved  Capability  HI  Program 
(Report  No.  D-2004-1 1 3) 


We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  in  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly.  In 
response  to  the  final  report,  we  request  that  the  Deputy  Assistant  Secretary  of  the  Navy 
(Air  Programs),  Office  of  the  Assistant  Secretary  of  the  Navy  (Research,  Development 
and  Acquisition)  reconsider  his  position  on  Recommendations  C.  1 .  and  C.2.  and  provide 
additional  comments  by  September  30, 2004. 

If  possible,  please  send  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only)  to  Audam@dodig.osd.mil.  Copies  of  the  management  comments  must 
contain  the  actual  signature  of  the  authorizing  official.  We  cannot  accept  the  /  Signed  / 
symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send  classified  comments 
electronically,  they  must  be  sent  over  the  SECRET  Internet  Protocol  Router  Network 
(SIPRNET). 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  Mr.  Rodney  D.  Britt  at  (703)  604-9096  (DSN  664-9096)  or  Ms.  Melinda  M.  Oleksa  at 
(703)  604-9093  (DSN  664-9093).  The  team  members  are  listed  inside  the  back  cover. 

See  Appendix  G  for  the  report  distribution. 

By  direction  of  the  Deputy  Inspector  General  for  Auditing: 


T^-Mary  L.  Ugoj; 
Assistant  Inspector  General 
for  Acquisition  Management 


Office  of  the  Inspector  General  of  the  Department  of  Defense 
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(Project  No.  D2003AE-0190) 

Acquisition  of  the  EA-6B  Improved  Capability  III  Program 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Civil  service  and  military  managers 
involved  in  the  management,  support,  and  oversight  of  the  EA-6B  Improved  Capability 
(ICAP)  III  Program  should  read  this  report  because  it  discusses  acquisition  issues  that 
must  be  addressed  before  the  EA-6B  ICAP  III  Program  progresses  further  through  the 
acquisition  process. 

Background.  The  Services  use  the  airborne  electronic  attack  capability  to  suppress  and 
degrade  an  opposing  force’s  air  defense  and  communication  systems  with  airborne 
electronic  jamming  before  offensive  airborne  strikes.  The  Navy’s  EA-6B  Prowler 
(EA-6B)  is  the  only  DoD  platform  that  provides  the  Services  with  an  airborne  electronic 
attack  capability  and  must  be  able  to  suppress  and  degrade  current  and  future  threats 
through  2015.  The  EA-6B  ICAP  III  Program  will  provide  the  EA-6B  aircraft  with 
upgraded  selective-reactive  jamming  and  threat  emitter  geo-location  capabilities.  In 
June  2003,  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  approved  the  program  for  low-rate  initial  production.  The  Program 
Manager  for  the  EA-6B  (the  Program  Manager)  subsequently  awarded  the  prime 
contractor,  Northrop  Grumman  Integrated  Systems,  Bethpage,  New  York,  a  firm-fixed- 
price  modification  to  the  development  contract  for  low-rate  initial  production  of  10  ICAP 
III  system  kits  for  $102  million.  The  Navy  plans  to  acquire  35  ICAP  III  system  kits  for 
installation  on  35  EA-6B  aircraft.  As  of  June  2004,  the  ICAP  III  Program  costs  consisted 
of  an  estimated  $335  million  in  research,  development,  test  and  evaluation  funds, 

$458  million  in  procurement  funds,  and  $109  million  in  operations  and  support  funds  for 
a  total  program  cost  of  $902  million. 

Results.  The  Program  Manager  provided  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  with  incomplete  information  on  the  operational 
assessment  that  the  Commander,  Operational  Test  and  Evaluation  Force  prepared  in 
support  of  the  ICAP  III  Program  low-rate  initial  production  decision.  Further,  the 
Designated  Approving  Authority  for  the  Naval  Air  Systems  Command  issued  an  Interim 
Authority  to  Operate  that  was  needed  to  begin  the  operational  test  phase  of  the  ICAP  III 
information  system  before  the  information  system  security  requirements  were  complete. 
Finally,  the  Program  Manager  did  not  require  the  subcontractor,  through  the  prime 
contractor,  for  the  ICAP  III  Tactical  Jamming  System  Receiver  to  submit  updated 
reliability  prediction  data  needed  to  determine  the  best  support  strategy  for  the  receiver 
system.  As  a  result,  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  approved  the  program  for  low-rate  initial  production,  increasing  the  risk  that 
costly  retrofit  expenses  will  occur  to  correct  the  design  deficiencies  for  the  low-rate 
initial  production  systems  and  incurring  higher  than  expected  sustainment  costs  for  the 
receiver  when  the  ICAP  III  Program  becomes  operational.  Ensuring  the  resolution  of  the 
critical  operational  issues  and  identifying  fixes  for  the  additional  deficiencies  identified 
in  the  operational  assessment  will  increase  the  likelihood  that  the  ICAP  III  system  will 


perform  satisfactorily  and  enable  the  milestone  decision  authority  to  make  a  fully 
informed  full-rate  production  decision.  Completing  required  elements  of  the  Department 
of  Defense  Information  Technology  Security  Certification  and  Accreditation  Process 
before  issuing  authority  to  operate  will  provide  assurance  to  decision  makers  that  the 
ICAP  III  information  system  will  meet  information  assurance  requirements.  Performing 
analysis  and  corrective  action  on  the  ICAP  III  hardware  failures  will  enable  the  Program 
Manager  to  determine  the  most  effective  method  for  meeting  future  ICAP  III  sustainment 
needs.  See  the  Findings  section  of  the  report  for  detailed  recommendations. 

The  Program  Manager’s  execution  of  the  management  control  program  was  generally 
adequate.  However,  he  should  include  a  review  of  test  and  evaluation  results, 
information  assurance,  and  system  reliability  failure  reviews  in  his  self-assessments. 

Management  Comments.  We  received  comments  from  the  Deputy  Assistant  Secretary 
of  the  Navy  (Ah'  Programs),  who  responded  for  the  Assistant  Secretary  of  the  Navy 
(Research,  Development,  and  Acquisition);  the  Commander,  Naval  Air  Systems 
Command;  and  the  Program  Manager  for  the  EA-6B.  Although  the  Deputy  Assistant 
Secretary  nonconcurred  with  the  finding  on  the  readiness  of  the  EA-6B  ICAP  III  for  low- 
rate  initial  production,  he  stated  that  the  Commander,  Operational  Test  and  Evaluation 
Force  would  determine  whether  the  22  critical  operational  issues  were  satisfactorily 
resolved  as  part  of  the  dedicated  operational  test  and  evaluation.  The  Deputy  Assistant 
Secretary  also  stated  that  the  Assistant  Secretary  of  the  Navy  (Research,  Development, 
and  Acquisition)  had  established  the  criteria  to  demonstrate  that  the  EA-6B  Improved 
Capability  III  Program  was  operationally  effective  and  suitable  before  approval  for  full- 
rate  production.  Also,  the  Deputy  Assistant  Secretary  agreed  that  the  Naval  Air  Systems 
Command  would  implement  recommended  actions  to  complete  required  elements  of  the 
Defense  Information  Technology  Security  Certification  and  Accreditation  Process  to 
ensure  that  the  EA-6B  ICAP  III  meets  information  assurance  requirements.  Further,  the 
Deputy  Assistant  Secretary  stated  that  the  Naval  Inventory  Control  Point  would  revise 
the  reliability  predictions  for  the  Tactical  Jamming  System  Receiver,  using  reliability 
data  from  the  dedicated  technical  and  operational  tests,  to  determine  the  most  effective 
method  for  meeting  ICAP  III  sustainment  needs.  The  Deputy  Assistant  Secretary  did  not 
agree  that  the  subcontractor  for  the  Tactical  Jamming  System  Receiver  needed  to  cross- 
reference  actions  taken  to  close  each  failure  identified  in  the  subcontractor’s  failure 
logbook  or  obtain,  through  the  prime  contractor,  an  updated  failure  modes  and  effects  and 
critical  analysis  document  required  by  the  Systems  Engineering  Management  Plan.  See 
the  Findings  section  of  the  report  for  a  discussion  of  management  comments  and  the 
Management  Comments  section  of  the  report  for  the  complete  text  of  the  comments. 

Audit  Response.  In  response  to  the  final  report,  we  request  that  the  Deputy  Assistant 
Secretary  of  the  Navy  (Air  Programs)  reconsider  his  position  on  requiring  that  the 
subcontractor  for  the  Tactical  Jamming  System  Receiver  document,  through  the  prime 
contractor,  the  failure  logbook  as  required  and  update  the  failure  modes  and  effects  and 
critical  analysis  document.  We  request  that  the  Deputy  Assistant  Secretary  provide  the 
additional  comments  by  September  30,  2004. 
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Background 


The  Services  use  the  Navy’s  airborne  electronic  attack  capability  to  suppress  and 
degrade  an  opposing  force’s  air  defense  and  communication  systems  with  airborne 
electronic  jamming  before  offensive  airborne  strikes.  The  Navy  defines  an 
airborne  electronic  attack  as  a  radio  frequency  system,  hosted  on  an  airborne 
platform,  which  limits  the  attack  of  air  breathing  (non- space)  free  flight  vehicles. 
The  Navy’s  EA-6B  Prowler  (EA-6B)  is  the  only  DoD  platform  that  provides  the 
Services  with  an  airborne  electronic  attack  capability  and  must  be  able  to  suppress 
and  degrade  current  and  future  threats  through  2015.  The  EA-6B  aircraft  became 
the  only  DoD  tactical  jamming  aircraft  when  the  Ah’  Force  terminated  the  EF-1 1 1 
Program  in  1995.  Accordingly,  the  Navy  EA-6B  aircraft  is  now  considered  a 
national  asset. 

Airborne  Electronic  Attack  Requirements.  The  Navy  operates  19  EA-6B 
squadrons;  1 1  are  carrier-based  squadrons,  4  are  joint  Air  Force  and  Navy 
expeditionary  squadrons,  and  4  are  Marines  Corps  expeditionary  squadrons.1  The 
Navy  and  Air  Force  have  shared  the  4  expeditionary  squadrons  since  the  Air 
Force  terminated  the  EF-1 1 1  Program.  As  of  June  2004,  the  Navy  has  119  EA-6B 
aircraft,  of  which  48  are  in  repair,  30  are  assigned  to  Navy  aircraft  carrier 
squadrons,  9  are  assigned  to  Navy  expeditionary  squadrons,  15  are  assigned  to 
Marine  Coips  expeditionary  squadrons,  13  are  assigned  for  fleet  replacement2, 
and  4  are  assigned  for  reserve  training  and  testing.  In  2015,  the  Navy  plans  to 
retire  the  EA-6B  aircraft  and  discontinue  the  expeditionary  squadrons  after  it 
acquires  90  EA-18G  replacement  aircraft  to  support  its  earner-based  squadrons. 
As  programmed,  the  Navy  will  not  support  requirements  for  future  airborne 
electronic  attack  missions  of  the  Air  Force  and  Marine  Coips. 

EA-6B  Program.  The  first  EA-6B  production  aircraft  was  introduced  in  1971, 
and  the  last  Prowler  was  delivered  in  1991.  Since  1971,  the  Navy  upgraded  the 
EA-6B  with  the  Expanded  Capability,  the  Improved  Capability  (ICAP)  I  and  II, 
and  made  Block  82,  86,  89,  and  89A  configuration  upgrades.  The  ICAP  III,  the 
Multifunctional  Information  Distribution  System  Fink- 16  network,  and  the  Joint 
Mission  Planning  System  programs  are  the  next  planned  major  modifications  to 
the  EA-6B  aircraft. 

ICAP  III  Program.  The  ICAP  III  Program,  an  Acquisition  Category  II  program, 
will  upgrade  the  software  and  hardware  in  the  EA-6B  aircraft  to  provide  a 
selective-reactive  jamming  capability  in  a  wider  frequency  range  and  a  threat 
emitter  geo-location  capability.  The  ICAP  III  Program  includes  the  design, 
development,  and  testing  of  a  new  upgraded  tactical  jamming  subsystem,  a 
tactical  information  subsystem,  and  a  tactical  display  subsystem.  The  EA-6B 
aircraft’s  performance  will  be  improved  when  the  existing  aircraft’s  surveillance 
receiver,  display  and  loader,  and  recorder  subsystems  are  replaced  with  more 
reliable  and  accurate  subsystems.  See  Appendix  C  for  a  full  description  of  the 
ICAP  III  Program  subsystems  that  will  upgrade  the  EA-6B  aircraft. 


A 1 1  expeditionary  squadron  is  an  armed  force  organized  to  accomplish  a  specific  objective  in  a  foreign 
country. 

2Fleet  Replacement  is  Navy  squadron  that  trains  pilots  to  operate  EA-6B  aircraft. 
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In  March  1998,  the  Navy  competitively  awarded  a  cost-plus-award-fee  contract  to 
Northrop  Grumman  Integrated  Systems,  Bethpage,  New  York,  for  the 
development  of  the  ICAP  III.  As  of  June  2004,  the  development  contract  price 
was  $235  million,  hi  June  2003,  the  Program  Manager  awarded  Northrop 
Grumman  a  firm-fixed-price  modification  to  the  development  contract  for  low- 
rate  initial  production  of  10  ICAP  III  system  kits  for  $102  million.  The  Navy 
plans  to  acquire  35  ICAP  III  kits  to  integrate  into  35  EA-6B  aircraft  at  an 
estimated  cost  of  $335  million  for  research,  development,  test  and  evaluation 
funds;  $458  million  for  procurement  funds;  and  $109  million  in  operations  and 
support  funds  for  a  total  program  cost  of  $902  million. 

Test  asset  unavailability  and  hardware  and  software  problems  caused  the  ICAP  III 
Program  to  experience  significant  schedule  delays  and  breach  schedule 
parameters  in  the  acquisition  program  baseline  agreement,  which  was  approved  in 
October  2003.  In  April  2004,  the  Commander,  Operational  Test  and  Evaluation 
Force  (COMOPTEVFOR)  approved  the  ICAP  III  Program  to  start  operational 
testing.  Because  of  the  schedule  delays,  however,  the  Program  Manager  was 
unable  to  obligate  and  expend  the  ICAP  III  procurement  funds  budgeted  for 
FY  2004.  In  June  2004,  the  Program  Manager  was  revising  the  agreement  to 
reflect  a  more  realistic  program  schedule  for  the  full-rate  production  decision. 
Accordingly,  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  had  not  yet  approved  a  revised  schedule  for  integrating  the  ICAP  III 
into  the  initial  EA-6B  aircraft. 

Within  the  Naval  Air  Systems  Command,  the  Program  Management  Activity-234, 
under  the  supervision  of  the  Navy  Program  Executive  Officer  for  Tactical  Aircraft 
Programs,  manages  the  ICAP  III  and  the  other  EA-6B  aircraft  upgrade  programs. 
The  Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  is 
the  milestone  decision  authority  for  all  EA-6B  aircraft  Acquisition  Category  II 
upgrades,  including  the  ICAP  III  Program. 


Objective 


The  audit  objective  was  to  evaluate  the  overall  acquisition  management  of  the 
EA-6B  ICAP  III  Program.  Specifically,  we  evaluated  whether  management  was 
cost-effectively  readying  the  program  for  the  production  phase  of  the  acquisition 
process.  We  also  evaluated  the  management  control  program  as  it  related  to  the 
audit  objective.  See  Appendix  A  for  a  discussion  of  the  scope  and  methodology 
of  the  review,  the  review  of  the  management  control  program,  and  prior  coverage 
related  to  the  audit  objective. 
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A.  Readiness  for  Low-Rate  Initial 
Production 

The  Program  Manager  provided  the  Assistant  Secretary  of  the  Navy 
(Research,  Development,  and  Acquisition)  with  incomplete  information 
on  the  operational  assessment  of  the  ICAP  III  Program  that  the 
Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) 
prepared  in  support  of  the  low-rate  initial  production  decision. 

Specifically,  the  Program  Manager  did  not  provide  the  details  for  the 
following  operational  test  results: 

•  The  high  false  emitter  display  ratio3  substantially  reduced  overall 
mission  effectiveness  by  overloading  the  aircrew’s  displays  with 
erroneous  symbols  and  caused  significant  problems  in  the  proper 
use  of  reactive  assignments. 

•  The  lack  of  a  fully  functional  keypad  prevented  the  test  aircrew 
from  using  the  ICAP  III  weapon  system  effectively. 

•  The  Tactical  EA-6B  Mission  Planning  System  erroneously 
displayed  data  (lost  or  frozen  data)  because  of  frequent  system 
lockups,  which  presented  major  obstacles  to  effective  pre-flight 
mission  planning  and  post-flight  analysis. 

•  Fifty  additional  performance  deficiencies  for  the  ICAP  III  systems 
required  correction  because  they  detracted  from  the  overall  EA-6B 
efficiency  or  operator  situational  awareness. 

The  Program  Manager  did  not  provide  the  COMOPTEVFOR  operational 
assessment  to  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  as  required  for  Navy  program  milestone 
decision  meetings  because  the  Program  Manager  limited  his  presentation 
of  the  COMOPTEVFOR  test  results  to  their  conclusion  that  the  ICAP  III 
was  potentially  operationally  effective  and  suitable,  the  ratings  for  the 
critical  operational  issues,  and  a  listing  of  the  50  additional  deficiencies. 
The  briefing  did  not  describe  how  the  deficiencies  affected  operational 
effectiveness  and  suitability.  As  a  result,  the  Assistant  Secretary  of  the 
Navy  (Research,  Development,  and  Acquisition)  approved  the  Program 
Manager’s  request  in  June  2003  to  procure  10  ICAP  III  systems  for  low- 
rate  initial  production,  and  the  Navy  increased  the  risk  that  it  will  incur 
costly  retrofit  expenses  to  correct  the  design  deficiencies  for  those  systems 
at  the  completion  of  the  dedicated  operational  test  and  evaluation  phase. 


3The  percentage  of  threat  emitter  symbols  displayed  to  the  operators  that  were  not  a  threat  signal  of 
interest. 
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Operational  Assessment 


To  assess  the  ICAP  III  system,  the  COMOPTEVFOR  tested  two  modified  EA-6B 
aircraft  for  a  total  of  12  test  missions  and  4  unit-level  training  flights,  hi  the 
operational  assessment,  the  COMOPTEVFOR  reported  that  the  ICAP  III  met  or 
exceeded  all  performance  threshold  values  with  the  exception  of  system 
reliability.  For  the  22  critical  operational  test  issues,  the  COMOPTEVFOR  rated 
3  issues  as  high  risk  and  another  9  issues  as  moderate  risk  on  achieving  program 
performance  requirements.  In  summary,  the  COMOPTEVFOR  stated  that  the 
demonstrated  capability  of  the  ICAP  III  equaled  or  exceeded  the  performance 
level  of  the  preceding  ICAP  II  system  and  that  the  overall  ICAP  III  system 
performance  was  low  risk.  The  COMOPTEVFOR  recommended  that  the  specific 
performance  deficiencies  identified  in  the  operational  assessment  be  corrected 
before  the  start  of  the  dedicated  operational  test  phase.  The  table  lists  the  critical 
operational  effectiveness  and  suitability  issues  and  the  corresponding  level  of  risk 
that  the  COMOPTEVFOR  assigned. 


Critical  Operational  Issue  Ratings 
Unclassified 


Critical  Issues-  Effectiveness  Risk  Rating1 

Detection  Red 

Jammer  Assignment  Yellow 

Identification  Yellow 

Geo-Location  Yellow 

Crew  Vehicle 

Aircrew  Factors  Red 

Integration  of  USQ- 1 13  and  MATT  IDM2  White 

Overall  System  Performance  Green 

Targeting 

High  Speed  Antiradiation  Missile  White 

Jammer  Steering  Green 

Tactics  Green 

Survivability  Green 

Joint  Interoperability  White 

Mission  Planning  and  Analysis  Red 

System  Reprogramability  Yellow 
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Critical  Issues  -Suitability 


Reliability  Yellow 

Maintainability  Yellow 

Availability  Green 

Logistic  supportability  White 

Compatibility  Green 

Interoperability  White 

Training  Green 

Human  factors  Yellow 

Safety  Yellow 

Documentation  Yellow 


^ed  =  high  risk;  yellow  =  medium  risk  =  white  risk;  green  =  low  risk;  white  =  not  evaluated, 
2Multimission  Advanced  Tactical  Terminal  Improved  Data  Modem. 


Policy  for  Low-Rate  Initial  Production  Decision 


DoD  established  policy  for  translating  mission  needs  and  technology 
opportunities  that  are  based  on  approved  mission  need  statements  and 
requirements  documents  into  stable,  affordable,  and  well-managed  acquisition 
programs.  This  transition  process  involves  two  decision  points:  approval  to  enter 
into  low-rate  initial  production  and  approval  to  enter  into  full-rate  production. 

The  low-rate  initial  production  decision  is  critical  because  it  stalls  the  contractor’s 
production  line.  DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition 
System,”  May  12,  2003,  establishes  mandatory  procedures  for  testing  ah  DoD 
acquisition  programs  before  production  decision  points.  DoD  Manual  4245. 7-M, 
“Transition  from  Development  to  Production,”  September  1985,  provides 
guidance  on  minimizing  risks  associated  with  transitioning  acquisition  programs 
from  development  to  production.  Secretary  of  the  Navy  Instruction  5420. 188E, 
“Acquisition  Category  Program  Decision  Process,”  December  11, 1997, 
establishes  Navy  policy  for  making  acquisition  program  decisions. 

DoD  Instruction  5000.2.  DoD  Instruction  5000.2  states  that  the  designated 
milestone  decision  authority  shah  base  the  decision  for  low-rate  initial  production 
on  acceptable  performance  in  developmental  test  and  evaluation  and  on  an 
operational  assessment,  acceptable  interoperability,  and  acceptable  operational 
supportability. 

DoD  Manual  4245.7-M.  DoD  Manual  4245. 7-M  identifies  risk-reduction 
activities  for  acquisition  managers  in  design,  testing,  and  production  in  support  of 
each  sequential  acquisition  program  milestone.  The  Manual  states  that  the 
acquisition  program’s  failure  to  perform  will  in  one  of  the  design,  test,  and 
production  processes  will  often  result  in  a  failure  to  do  well  in  ah  areas  and  cause 
programs  to  become  high  risk.  By  not  accomplishing  risk-reduction  activities,  the 
Manual  states  that  equipment  will  be  deployed  later  and  at  a  far  greater  cost  than 
originally  planned. 
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Secretary  of  the  Navy  Instruction  5420.188E.  Secretary  of  the  Navy 
Instruction  5420.1 8 8E  requires  program  managers  to  provide  the  operational 
assessment  report  that  the  COMOPTEVFOR  prepares  to  the  Assistant  Secretary 
of  the  Navy  (Research,  Development,  and  Acquisition)  to  support  a  low-rate 
initial  production  milestone  review  for  acquisition  programs. 


Reporting  the  Operational  Assessment  Results  for  the  Navy 
Program  Decision  Meeting 


The  Program  Manager  provided  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  with  incomplete  information  on  the  operational 
assessment  of  the  ICAP  III  Program  that  COMOPTEVFOR  prepared  in  support 
of  the  low-rate  initial  production  decision.  Specifically,  the  Program  Manager  did 
not  provide  the  details  from  the  operational  assessment  on  the  three  high-risk, 
critical  operational  issues;  that  is,  major  deficiencies  with  ICAP  III  detection 
capabilities,  crew-vehicle  interface,  and  mission  planning  and  analysis.  In  the 
briefing  backup  slides  for  the  Navy  program  decision  meeting,  the  Program 
Manager  reported  that  the  operational  assessment  also  identified  50  additional 
deficiencies  that  detracted  from  overall  system  efficiency  and  operator  awareness. 

High-Risk,  Critical  Operational  Issues.  An  explanation  of  the  three  high-risk, 
critical  operational  issues  follows. 

Detection.  The  COMOPTEVFOR  reported  that  the  ICAP  III  did  not 
demonstrate  the  potential  for  providing  accurate  and  timely  detection  and  display 
of  threat  emitters  in  single  and  multiple  emitter  environments  and  in  jamming  and 
non-jamming  environments.  The  report  stated,  although  test  emitters  had  been 
detected  and  the  system  had  demonstrated  the  ability  to  rapidly  display  the 
emitters  in  jamming  and  non-jamming  environments,  the  false  emitter  display 
ratio  was  exceeding  high.  The  COMOPTEVFOR  also  stated  that  in  the  high  false 
emitter  display  ratio  cluttered  the  system  display  with  erroneous  information  and 
would  cause  system  operators  to  increase  their  workload  and  decrease  their 
situational  awareness. 

To  assess  the  system’s  detection  ability,  the  COMOPTEVFOR  tested  the  ICAP 
III  against  24  tiueat  emitters.  The  COMOPTEVFOR  reported  that  the  ICAP  III 
demonstrated  a  false  emitter  display  ratio  of  greater  than  80  percent  for  12  threat 
emitters;  a  false  emitter  display  ratio  of  between  50  and  80  percent  for  8  threat 
emitters;  and  a  false  emitter  display  ratio  of  less  than  50  percent  for  4  threat 
emitters.  Further,  the  COMOPTEVFOR  stated  that  the  false  emitter  display  ratio 
had  been  extreme  against  one  of  the  24  threat  emitters.  In  the  extreme  example, 
the  COMOPTEVFOR  reported  that,  during  a  1-hour  period,  1,002  of  1,012  threat 
emitter  symbols  displayed  to  the  operators  were  not  associated  with  the  actual 
threat  emitter.  The  COMOPTEVFOR  concluded  that  if  the  operators’  display 
became  flooded  with  false  emitter  symbols  at  that  magnitude,  operator 
effectiveness  against  that  threat  emitter  would  be  severely  reduced.  Additionally, 
the  COMOPTEVFOR  reported  that,  the  aircrews  that  tested  the  ICAP  III 
expressed  very  low  confidence  in  the  authenticity  of  the  threat  emitter  symbols 
displayed  to  them  during  the  test  missions. 
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Crew-Vehicle  Interface.  The  COMOPTEVFOR  reported  that  the  crew- 
vehicle  interface  capability  of  the  ICAP  III  did  not  demonstrate  the  potential  for 
supporting  aircrew  situational  awareness  and  mission  accomplishment.  To 
measure  the  crew-vehicle  interface  capability,  the  COMOPTEVFOR  identified 
aircrew  factors  and  the  need  to  assess  the  integration  of  the  USQ-1 13 
Communications  Jamming  System  and  the  Multi-mission  Advanced  Tactical 
Terminal  Improved  Data  Modem.  The  COMOPTEVFOR  rated  the  risk  of  the 
crew-vehicle  interface  capability  of  the  ICAP  III  as  high  because  the  system  did 
not  have  a  fully  functional  alphanumeric  keyboard,  which  represented 
degradation  in  capability  from  what  was  available  in  the  ICAP  II  SYSTEM.  The 
COMOPTEVFOR  did  not  assess  the  USQ-1 13  Communications  Jamming  System 
and  the  Multi-mission  Advanced  Tactical  Terminal  Improved  Data  Modem 
because  those  items  had  yet  to  be  integrated  into  the  ICAP  III  aircraft. 

Mission  Planning  and  Analysis.  The  COMOPTEVFOR  reported  that 
version  7/8  of  the  Tactical  EA-6B  Mission  Planning  System,  the  current  mission 
planning  system  for  the  ICAP  III,  did  not  demonstrate  the  potential  for  supporting 
the  ICAP  III  mission.  The  COMOPTEVFOR  stated  that  the  major  obstacle  to 
effective  mission  planning  and  post-flight  mission  analysis  for  the  operational 
assessment  was  the  instability  of  the  Tactical  EA-6b  Mission  Planning  System. 
The  report  documented  instances  where  the  Tactical  EA-6B  Mission  Planning 
System  failed,  resulting  in  a  halt  to  mission  planning  during  the  assessment  and 
causing  the  operator  to  redo  previously  completed  work  because  of  lockups  or 
erroneously  displayed  or  deleted  data.  Furthermore,  the  COMOPTEVFOR  stated 
that  the  functionality  of  the  Tactical  EA-6B  Mission  Planning  System  was 
inadequate.  Specifically,  the  report  stated  that  the  type  of  information  and  the 
method  of  entering  information  were  not  user  friendly.  The  COMOPTEVFOR 
also  reported  that  most  mission  information  produced  by  the  Tactical  EA-6B 
Mission  Planning  System  was  either  indecipherable  or  insufficient  for  proper 
briefing.  The  report  further  stated  that  many  of  the  current  threat  emitter  library 
loads  on  the  Tactical  EA-6B  Mission  Planning  System  did  not  adequately  support 
the  improved  signal  identification  capabilities  of  the  ICAP  III.  In  conclusion,  the 
COMOPTEVFOR  reported  that  the  excessive  number  of  malfunctions 
significantly  reduced  the  operator’s  confidence  in  the  system  and  drastically 
reduced  the  operator’s  mission  planning  effectiveness. 

Recommendations  from  Operational  Assessment.  The  COMOPTEVFOR 
recommended  that  the  following  three  deficiencies  related  to  the  high-risk,  critical 
operational  issues  be  corrected  and  verified  during  the  dedicated  operational  test 
and  evaluation: 

•  Investigate  and  correct  the  high  false  emitter  display  ratio. 

•  Correct  alphanumeric  keyboard  disfunctionality. 

•  Investigate  and  correct  the  unreliable  Tactical  EA-6B  Mission 
Planning  System  version  7/8. 

At  the  Navy  program  decision  briefing  to  the  Assistant  Secretary  of  the  Navy 
(Research,  Development,  and  Acquisition),  the  Program  Manager  listed  the 
50  additional  deficiencies  as  having  minor  to  moderate  risk  on  achieving  program 
performance  requirements.  Although  the  50  additional  deficiencies  were 
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identified  as  not  critical  to  mission  accomplishment,  the  COMOPTEVFOR 
indicated  that  they  should  be  corrected  and  verified  during  the  dedicated 
operational  test  and  evaluation  phase  because  they  detracted  from  the  overall  EA- 
6B  efficiency  and  operator  situational  awareness.  See  Appendix  D  for  the 
complete  description  of  the  50  additional  deficiencies. 


Requirement  for  Navy  Program  Decision  Meetings 


The  Program  Manager  did  not  provide  the  COMOPTEVFOR  operational 
assessment  report  to  the  Assistant  Secretary  of  the  Navy  (Research,  Development, 
and  Acquisition)  as  required  in  support  of  Navy  program  decision  meetings. 
Instead,  the  EA-6B  Program  Manager  limited  his  presentation  of  the 
COMOPTEVFOR  test  results  to  their  conclusions  that  the  EA-6B  ICAP  III  was 
potentially  operational  effective  and  suitable,  ratings  for  the  critical  operational 
issues,  and  the  backup  slides  listing  the  50  additional  deficiencies. 

Navy  Program  Decision  Meeting.  In  June  2003,  the  Program  Manager  briefed 
the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  on 
the  readiness  of  the  ICAP  III  to  enter  low-rate  initial  production.  The  Program 
Manager  stated  that  the  program  had  met  the  exit  criteria  that  the  Assistant 
Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  established  for 
entry  into  the  low-rate  initial  production  phase  of  the  acquisition  process.  The 
Program  Manager  detailed  that  the  COMOPTEVFOR  had  determined  that  the 
ICAP  III  system  was  potentially  operationally  effective  and  suitable.  He  also 
stated  that  the  COMOPTEVFOR  rated  12  of  the  22  critical  operational  issues  for 
effectiveness  and  suitability  at  moderate  to  high  risk  of  achieving  the  system 
capabilities  needed  for  the  EA-6B  to  accomplish  its  mission.  However,  the 
Program  Manager  did  not  provide  the  milestone  decision  authority  with  sufficient 
detail  on  the  12  critical  operational  issues  or  descriptions  of  the  50  additional 
deficiencies  to  ensure  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  could  make  an  informed  decision  on  the  readiness 
of  the  ICAP  III  to  enter  low-rate  initial  production  and  to  begin  the  dedicated 
operational  test  phase.  COMOPTEVFOR  representatives  were  invited,  but  were 
unable  to  attend  the  ICAP  III  Navy  program  meeting  for  making  the  low-rate 
initial  production  decision. 


Conclusion 


The  Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition) 
approved  the  Program  Manager’s  request  in  June  2003  to  procure  10  ICAP  III 
systems  for  low-rate  initial  production.  As  a  result,  the  Navy  increased  the  risk 
that  it  will  incur  costly  retrofit  expenses  to  correct  the  design  deficiencies  for 
those  systems  at  the  completion  of  the  dedicated  operational  test  and  evaluation 
phase. 

Until  the  Navy  produces  the  follow-on  EA-18G  aircraft,  the  EA-6B  is  the  only 
DoD  aircraft  that  can  perform  airborne  electronic  attack  operations.  Because  the 
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ICAP  III  system  will  be  the  baseline  capability  for  the  EA-18G,  it  is  particularly 
important  that  the  ICAP  III  system  be  thoroughly  tested  and  can  demonstrate  the 
satisfaction  of  operational  performance  requirements  before  approval  for  full-rate 
production.  To  improve  the  likelihood  that  the  EA-6B  ICAP  III  will  perform 
satisfactorily  during  the  dedicated  operational  test  and  evaluation  phase,  we 
believe  that  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  should  analyze  and  satisfactorily  resolve  the  high-risk  areas 
identified  during  developmental  testing  for  the  ICAP  III,  as  well  as  the 
50  additional  deficiencies  that  detracted  from  overall  efficiency  or  operator 
situational  awareness  identified  during  developmental  testing  before  the 
conclusion  of  the  dedicated  operational  test  and  evaluation  phase. 


Management  Comments  on  the  Finding  and  Audit  Responses 


Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
Office  of  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  nonconcurred  with  the  finding,  stating  that  the  COMOPTEVFOR 
delivered  the  operational  assessment  report  to  the  Office  of  the  Assistant 
Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  in  June  2003. 
Furthermore,  the  Deputy  Assistant  Secretary  stated  that  the  Program  Manager 
addressed  details  of  the  operational  assessment  results  and  provided  backup 
material  at  the  Navy  Program  Decision  Meeting.  Specifically,  he  stated  that  the 
backup  material  discussed  the  four  operational  mission  failures  that  occurred 
during  the  operational  assessment  test  and  the  plans  and  the  status  of  the 
corrective  actions.  Additionally,  the  Deputy  Assistant  Secretary  stated  that  the 
backup  material  identified  and  discussed  the  corrections  for  the  high  and 
moderate  suitability  risks  and  identified  all  minor  critical  operational  issues. 

Audit  Response.  Based  on  discussions  with  a  representative  from  the  Office  of 
the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition), 
we  acknowledge  that  the  COMOPTEVFOR  delivered  a  copy  of  the  assessment 
report  to  the  Office  of  the  Assistant  Secretary.  However,  the  representative  stated 
that  the  Assistant  Secretary  did  not  see  the  operational  assessment  report  before 
the  Navy  Program  Decision  Meeting,  nor  did  the  Program  Manager  provide  the 
Assistant  Secretary  with  a  copy  of  the  report  at  the  meeting.  As  stated  in  the 
report,  the  briefing  charts  and  backup  material  that  the  ICAP  III  Program 
Manager  presented  at  the  Navy  Program  Decision  Meeting  did  not  provide 
evidence  of  the  extent  of  the  operational  effectiveness  and  suitability 
shortcomings  identified  in  the  operational  assessment  report  that  were  discussed. 
If  the  milestone  decision  authority  had  been  fully  aware  of  the  extent  of  EA-6B 
ICAP  III  operational  performance  problems  identified  in  the  operational 
assessment  report,  he  may  not  have  made  the  decision  to  approve  low-rate  initial 
production.  Had  the  Program  Manager  provided  the  report  at  the  Navy  Program 
Decision  Meeting  as  required,  there  would  be  no  doubt  as  to  whether  the 
milestone  decision  authority  was  fully  informed  of  the  extent  of  the  operational 
effectiveness  and  suitability  shortcomings  of  the  EA-6B  ICAP  III  when  making 
the  low-rate  initial  production  decision. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


A.l.  We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  obtain  the  Commander,  Operational  Test 
and  Evaluation  Force’s  operational  assessment  for  the  EA-6B  Improved 
Capability  III  and  not  approve  full-rate  production  until  the  Commander, 
Operational  Test  and  Evaluation  Force  determines  the  satisfactory 
resolution  of  the  22  critical  operational  issues. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
responding  for  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition),  nonconcurred,  stating  that  as  part  of  the  preparation  for  the 
dedicated  operational  test  and  evaluation,  the  EA-6B  ICAP  III  Program  Office 
addressed  each  operational  assessment  item  to  ensure  that  improvements,  where 
warranted,  were  included  in  the  product  submitted  to  COMOPTEVFOR  for 
testing.  He  further  stated  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  established  in  the  acquisition  decision 
memorandum  the  requirement  that  COMOPTEVFOR  determine  that  the  EA-6B 
ICAP  III  Program  was  operationally  effective  and  suitable  before  the  full-rate 
production  decision  would  be  made.  The  Deputy  Assistant  Secretary  also  stated 
that  the  results  of  the  dedicated  operational  test  and  evaluation  should  take 
precedence  over  the  results  contained  in  the  operational  assessment  report  in 
making  the  full-rate  production  decision. 

Audit  Response.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
comments  were  responsive  to  the  intent  of  Recommendation  A.l.  The 
Commander,  Operational  Test  and  Evaluation  Force  will  revisit  and  determine  the 
satisfactory  resolution  of  the  22  critical  operational  issues  as  part  of  the  dedicated 
operational  test  and  evaluation  that  is  planned  before  the  full-rate  production 
decision.  We  agree  that  the  results  of  the  dedicated  operational  test  and 
evaluation  should  take  precedence  over  the  results  contained  in  the  operational 
assessment  report.  The  satisfactory  resolution  of  the  22  critical  operational 
issues,  however,  should  be  key  in  the  decision  of  the  Assistant  Secretary  of  the 
Navy  (Research,  Development,  and  Acquisition)  to  allow  the  EA-6B  ICAP  III 
Program  to  proceed  into  full-rate  production. 

A. 2.  We  recommend  that  the  Program  Manager  for  the  EA-6B  analyze  and 
identify  fixes  for  the  50  additional  deficiencies  identified  by  the  Commander, 
Operational  Test  and  Evaluation  Force  in  the  operational  assessment  to 
increase  the  likelihood  that  the  ICAP  III  will  perform  satisfactorily  before 
concluding  the  dedicated  operational  test  and  evaluation  phase  of  the 
acquisition  process. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
nonconcured,  stating  that  the  Program  Manager  for  the  EA-6B  was  already 
addressing  the  major  and  minor  deficiencies  identified  in  operational  assessment 
report.  He  stated  that  the  Program  Manager  established  a  process  to  resolve  the 
50  areas  of  risk  cited  in  the  operational  assessment  report  and  to  prioritize  their 
resolution.  The  Deputy  Assistant  Secretary  stated  that  resolution  ranged  from 
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immediate  action  to  removing  the  requirement  in  a  future  requirement  update. 
Additionally,  he  stated  that  the  COMPOPTEVFOR  identifies  and  prioritizes 
deficiencies  but  the  program  sponsor  provides  funding  to  fix  the  deficiencies. 

Audit  Response.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
comments  were  responsive  to  the  intent  of  Recommendation  A.2. 
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B.  An  Interim  Authority  to  Operate 
Improved  Capability  III  Information 
Systems 

In  May  2003,  the  Designated  Approving  Authority  for  the  Naval  Air 
Systems  Command  issued  the  Program  Manager  an  Interim  Authority  to 
Operate  (IATO)  the  ICAP  III  information  system  without  requiring  the 
Program  Manager  for  the  EA-6B  aircraft  to  first  complete  the  verification 
and  validation  phases  of  the  Department  of  Defense  Information 
Technology  Security  Certification  and  Accreditation  Process  (DITSCAP), 
as  required.  As  a  result,  the  Program  Manager  began  operational  testing 
of  a  system  that  may  not  satisfy  system  information  assurance 
requirements. 


Policy  and  Guidance  for  Certifying  and  Accrediting  DoD 
Information  Systems 


DoD  established  policy  and  guidance  to  manage  the  certification  and 
accreditation  process  for  information  systems.  DoD  Instruction  5200.40, 
“Department  of  Defense  Information  Technology  Security  Certification  and 
Accreditation  Process  (DITSCAP),”  December  30,  1997,  and  DoD  Manual 
8510.1-M,  “Department  of  Defense  Information  Technology  and  Security 
Certification  and  Accreditation  Process  (DITSCAP)  Application  Manual,”  July 
2000,  establish  the  process  for  accrediting  DoD  computers,  systems,  and 
networks. 

DoD  Instruction  5200.40.  DoD  Instruction  5200.40  implements  policy,  assigns 
responsibilities,  and  prescribes  procedures  for  certifying  and  accrediting 
information  technology  in  DoD  systems.  The  Instruction  identifies  four  phases 
for  the  DITSCAP  to  certify  that  the  information  technology  system  meets  the 
accreditation  requirements  and  that  the  system  continues  to  meet  those 
requirements  throughout  the  system’s  life  cycle.  A  description  of  the  four-phase 
DITSCAP  follows. 

•  Definition  -  uses  collected  information  to  determine  the 
certification  level  of  the  system,  which,  in  turn,  determines  the 
level  of  effort  required. 

•  Verification  -  includes  activities  to  verify  system  compliance  with 
security  requirements  and  to  evaluate  vulnerabilities. 

•  Validation  -  ensures  that  the  fully  integrated  system  operates  in  a 
specified  computing  environment  with  an  acceptable  level  of  risk. 

•  Post  Accreditation  -  includes  activities  to  monitor  system 
management  and  operation  to  ensure  that  an  acceptable  level  of 
residual  risk  is  preserved. 
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DoD  Manual  8510.1-M.  DoD  Manual  8510.1-M  defines  information  assurance 
as  information  operations  that  protect  and  defend  information  and  information 
systems  by  ensuring  their  availability,  integrity,  authentication,  confidentiality, 
and  nonrepudiation.  Information  operations  include  providing  for  the  restoration 
of  information  systems  by  incorporating  protection,  detection,  and  reaction 
capabilities.  Further,  the  Manual  provides  guidance  for  program  managers  of 
information  systems  on  the  implementation  of  the  DITSCAP  and  defines  the 
responsibilities  of  the  Designated  Approving  Authority,  the  Certification 
Authority,  and  their  certification  and  accreditation  activities. 

Designated  Approving  Authority.  The  Designated  Approving  Authority 
is  the  official  with  the  authority  and  ability  to  evaluate  the  mission,  business  case, 
and  budgetary  needs  for  the  system  while  accounting  for  potential  security  risks. 
The  Designated  Approving  Authority  determines  the  acceptable  level  of  residual 
risk  and  approves  the  system  for  operation.  The  Designated  Approving  Authority 
for  the  EA-6B  ICAP  III  Program  is  a  representative  from  the  Information 
Assurance  Office  at  the  Naval  Air  Systems  Command. 

Certification  Authority.  The  Certification  Authority  provides  the 
technical  expertise  to  conduct  the  certification  of  the  system  throughout  the 
system’s  life-cycle  based  on  the  security  requirements  documented  in  the  System 
Security  Authorization  Agreement.  The  Certification  Authority  identifies  and 
assesses  the  risks  associated  with  operating  the  system  and  makes  an  accreditation 
recommendation  to  the  Designated  Approving  Authority.  The  Certification 
Authority  for  the  EA-6B  ICAP  III  Program  is  also  a  representative  from  the 
Information  Assurance  Office  at  the  Naval  Air  Systems  Command. 

Requirements  Described  in  System  Security  Authorization 
Agreement.  DoD  Manual  8510.1-M  lists  the  requirements  that  should  be 
described  in  a  System  Security  Authorization  Agreement  (the  Agreement).  The 
DoD  Manual  states  that  the  Agreement,  which  is  a  foimal  agreement  between  the 
Designated  Approving  Authority,  Certification  Authority,  user  representative,  and 
the  Program  Manager,  should  be  used  to  guide  and  document  the  results  of  the 
certification  and  accreditation  process.  The  Manual  further  states  that  the 
objective  of  the  Agreement  is  to  establish  an  evolving,  yet  binding,  agreement  on 
the  level  of  security  needed  before  beginning  system  development.  The  DoD 
Manual  also  states  that  the  Certification  Authority  must  analyze  the  system  and 
deteimine  the  degree  of  confidentiality,  integrity,  availability,  and  accountability 
required  for  the  system.  Based  on  this  analysis,  the  Certification  Authority 
recommends  a  certification  level  to  which  the  DITSCAP  certification  tasks  must 
be  performed  and  documents  this  certification  level  in  the  Agreement.  Once  the 
DITSCAP  process  is  complete,  the  DoD  Manual  states  that  the  Agreement 
becomes  the  baseline  security  configuration  document. 

Interim  Authority  to  Operate.  DoD  Manual  8510.1-M  also  establishes 
guidance  for  issuing  an  IATO.  The  DoD  Manual  specifies  the  need  to  identify  a 
Certification  Authority  and  a  Designated  Approving  Authority  for  each 
information  system  as  the  individuals  that  would  oversee  the  DITSCAP  and 
process  information  based  on  preliminary  results  of  a  system  security  evaluation. 
The  DoD  Manual  states  that  if  an  information  system  has  not  met  the 
requirements  stated  in  the  Agreement,  but  that  mission-critical  needs  require  the 
system  to  become  operational,  the  Certification  Authority  may  recommend  that 
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the  Designated  Approving  Authority  issue  an  IATO  after  the  validation  phase  of 
the  DITSCAP  with  the  understanding  that  the  noted  deficiencies  will  be  corrected 
within  a  set  timeframe.  The  DoD  Manual  states  that  the  deficiencies  must  be 
noted  in  the  Agreement  and  that  the  Certification  Authority,  Designated 
Approving  Authority,  Program  Manager,  and  user  representative  must  agree  to 
the  proposed  solutions,  schedule,  security  actions,  milestones,  and  maximum 
length  of  time  for  the  IATO. 


ICAP  III  Program  Information  Systems 

The  ICAP  III  Program  integrates  existing  EA-6B  aircraft  systems  and  system 
upgrades  to  accomplish  selective-reactive  jamming  and  threat  emitter 
geo-location  capabilities  that  will  enable  the  future  EA-6B  aircraft  to  more 
effectively  perform  its  airborne  electronic  attack  operations.  The  major 
information  system  upgrades  for  the  EA-6B  aircraft  include  the  Tactical  Jamming 
Subsystem  Receiver,  the  Multifunctional  Information  Distribution  System  to 
support  the  Link- 16  network,  the  Tactical  Display  Subsystem,  and  the  Data 
Storage  Memory  Unit. 


Issuing  an  IATO  For  the  ICAP  III  Program 


In  May  2003,  the  Designated  Approving  Authority  for  the  Naval  Air  Systems 
Command  issued  the  Program  Manager  for  the  EA-6B  aircraft  an  IATO  before 
the  Certification  Authority  verified  that  the  ICAP  III  information  system  security 
requirements  were  complete. 

Completeness  of  System  Security  Authorization  Agreement.  DoD 
Manual  8510.1-M  states  that  the  Certification  Authority  should  identify  the 
certification  level  needed  for  DITSCAP  certification  tasks  in  the  ICAP  III 
Program  Agreement.  The  ICAP  III  Certification  Authority,  however,  did  not 
document  the  certification  level  in  the  draft  Agreement.  Without  identifying  the 
certification  level  and  the  level  of  effort  needed  to  complete  the  DITSCAP 
certification  tasks,  the  Certification  Authority  did  not  have  information  needed  to 
effectively  perform  the  verification  and  validation  phases  for  the  ICAP  III 
Program. 

Status  of  the  ICAP  III  Program  DITSCAP.  In  May  2003,  the  Program 
Manager  for  the  EA-6B  aircraft  issued  an  Information  Assurance  Strategy  for  the 
ICAP  III  Program.  The  Information  Assurance  Strategy  stated  that  DITSCAP 
certification  procedures  in  DoD  Instruction  5200.40  and  DoD  Manual  8510.1-M 
were  being  applied  for  the  ICAP  III  Program,  that  the  ICAP  III  Program  was  in 
the  definition  phase  of  the  DITSCAP,  and  that  the  Program  Manager  had 
provided  the  initial  information  needed  to  complete  the  definition  phase  and 
would  provide  the  additional  information  needed  to  complete  the  Agreement  for 
final  accreditation  of  the  ICAP  III  information  system.  As  of  June  2004,  a 
representative  at  the  Naval  Air  Systems  Command  stated  that  the  program  was 
still  in  the  definition  phase  of  the  DITSCAP,  and  that  the  Certification  Authority 
was  reviewing  the  applicable  requirements  for  the  system  security  baseline.  The 
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representative  stated  that  once  the  review  was  complete,  the  Designated 
Approving  Authority  would  perform  system  verification  and  validation  of  those 
baseline  requirements. 


Requirements  for  Issuing  an  IATO 


The  Designated  Approving  Authority  did  not  require  the  Program  Manager  for 
the  EA-6B  aircraft  to  complete  the  verification  and  validation  phases  of  the 
DITSCAP  as  required  before  issuing  the  IATO.  The  IATO  was  issued  during  the 
definition  phase  of  the  DITSCAP  instead  of  after  the  validation  phase  of  the 
DITSCAP  as  required  by  DoD  Manual  8510. 1-M.  A  representative  at  the  Naval 
Air  Systems  Command  stated  that  the  recommendation  to  issue  the  IATO  for  the 
ICAP  III  Program  was  based  on  the  Certification  Authority’s  analysis  of  the  draft 
Agreement,  a  review  of  the  system  architecture,  and  the  identification  of  any 
associated  risks.  The  representative  also  stated  that  the  IATO  was  issued  to 
prevent  delay  of  the  scheduled  testing.  Before  issuing  the  IATO  for  the  ICAP  III 
information  system,  the  Designated  Approving  Authority  should  have  required 
the  Program  Manager  to  complete  the  verification  and  validation  phases  of  the 
DITSCAP  to  ensure  system  compliance  with  security  requirements  and  to 
validate  that  the  fully  integrated  system  could  operate  in  a  specified  computing 
environment  with  an  acceptable  level  of  risk. 


Conclusion 


As  a  result  of  the  Program  Manager’s  obtaining  the  IATO,  the  ICAP  III  Program 
entered  into  operational  testing  without  the  assurance  that  the  system  could  satisfy 
system  information  assurance  requirements.  To  ensure  that  disruption  to  mission 
operations  from  information  security  risks  have  been  mitigated,  system 
infoimation  assurance  requirements  must  be  satisfied  so  that  an  appropriate  level 
of  confidentiality,  integrity,  availability,  and  authentication  is  protected  for  the 
ICAP  III  infoimation  systems. 

Management  Comments  on  the  Finding  and  Audit  Response 


Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
responding  for  the  Certification  Authority  for  the  EA-6B  Improved  Capability  III 
Program,  Naval  Air  Systems  Command,  partially  concurred  with  the  finding, 
stating  that  the  Naval  Air  Systems  Command  Designated  Approving  Authority 
and  Certification  Authority  reviewed  the  IATO  and  the  process  followed  by  the 
Information  Assurance  Program  Officer  in  issuing  this  document.  Although  he 
agreed  that  the  DITSCAP  typically  requires  the  review  and  issuance  of  the  IATO 
after  completion  of  the  verification  and  validation  phase  of  a  program,  he  stated 
that  interpretation  of  DITSCAP  requirements  did  not  accurately  reflect  the 
practical  application  of  an  information  assurance  program.  The  Deputy  Assistant 
Secretary  stated  that  the  Information  Assurance  Program  Officer  met  with  the 
Program  Manager  to  address  overall  information  assurance  requirements,  to 
include  testing  the  ICAP  III  variant  on  a  test  platform.  He  also  stated  that  the 
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Information  Assurance  Program  Office  assessed  the  security  architecture  and  the 
design  and  determined  there  was  minimal  risk  of  unauthorized  disclosure, 
modification,  or  availability  of  National  Security  Information,  and  that  if  an 
unacceptable  level  of  risk  had  been  determined  during  the  review  process,  the 
Naval  Air  Systems  Command  Designated  Approving  Authority  would  not  have 
authorized  the  interim  authority  to  operate  for  the  test  and  evaluation  period. 

Audit  Response.  We  agree  that  the  Naval  Air  Systems  Command  Designated 
Approving  Authority  and  Certification  Authority  reviewed  the  IATO  and  the 
process  following  its  issuance.  However,  in  not  completing  the  verification  and 
validation  phases  of  the  DITSCAP  for  the  EA-6B  ICAP  III  Program  as  required 
to  ensure  compliance  with  security  requirements  and  to  validate  that  the  fully 
integrated  system  could  operate  in  a  specified  computing  environment  with  an 
acceptable  level  of  risk,  the  Program  Manager  could  not  provide  assurance  to 
COMOPTEVFOR  that  the  EA-6B  ICAP  III  Program  could  satisfy  information 
assurance  requirements  before  operational  testing. 

Recommendations,  Management  Comments,  and  Audit 
Responses 


B.l.  We  recommend  that  the  Certification  Authority  for  the  EA-6B 
Improved  Capability  III  Program,  Naval  Air  Systems  Command: 

a.  Complete  all  definition,  verification,  and  validation 
requirements  to  comply  with  the  DoD  Information  Technology  Security 
Certification  and  Accreditation  Process. 

b.  Document  the  certification  procedures  and  test  results  in  the 
EA-6B  Improved  Capability  III  System  Security  Authorization  Agreement. 

c.  Use  the  results  from  completing  the  DoD  Information 
Technology  Security  Certification  and  Accreditation  Process  to  recommend 
to  the  Designated  Approving  Authority  for  the  EA-6B  Improved  Capability 
III  whether  to  continue  granting  the  EA-6B  Program  Manager  an  Interim 
Authority  to  Operate. 

B.2.  We  recommend  that  the  Designated  Approving  Authority  for  the  EA- 
6B  Improved  Capability  III  Program,  Naval  Air  Systems  Command  base  the 
decision  to  continue  granting  the  EA-6B  Program  Manager  an  Interim 
Authority  to  Operate  on  the  recommendation  of  the  Certification  Authority 
for  the  EA-6B  Improved  Capability  III  Program. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
responding  for  the  Certification  Authority  and  the  Designated  Approving 
Authority  for  the  EA-6B  Improved  Capability  III  Program,  Naval  Air  Systems 
Command,  concurred  with  the  two  recommendations  and  stated  that  the 
Certification  Authority  planned  to  accomplish  the  actions  listed  in 
Recommendation  B.l.  and  would  make  his  recommendations  on  whether  to 
continue  granting  the  IATO  to  the  Designated  Approving  Authority. 

Audit  Response.  The  Deputy  Assistant  Secretary’s  comments  were  responsive  to 
the  two  recommendations. 
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C.  Use  of  Failure  Data  for  the  Improved 
Capability  III  Program  Hardware 

The  subcontractor  did  not  submit,  through  the  prime  contractor  for  the 
ICAP  III  Tactical  Jamming  System  Receiver,  updated  reliability 
prediction  data  needed  to  perform  a  cost-benefit  analysis  to  determine  the 
best  maintenance  and  logistical  support  strategy  for  the  receiver.  This 
condition  occurred  because  the  Program  Manager  did  not  direct  the  prime 
contractor  to  update  reliability  predictions  after  analyzing  and  resolving 
all  hardware  failures  identified  during  developmental  testing  or  to  retain 
documentation  of  corrective  actions  taken  to  reduce  the  frequency  of 
hardware  failures.  As  a  result,  the  Program  Manager  is  not  able  to 
accurately  predict  the  expected  reliability  of  the  Tactical  Jamming  System 
Receiver  and  may  incur  higher  than  expected  costs  to  maintain  and 
logistically  support  the  receiver  when  the  ICAP  III  Program  becomes 
operational. 


System  Reliability  Criteria  and  Guidance 


The  Interim  Defense  Acquisition  Guidebook  provides  program  managers  with 
guidance  to  use  to  sustain  system  reliability.  DoD  Manual  4245. 7-M,  “Transition 
from  Development  to  Production,”  September  1985,  provides  program  managers 
with  additional  guidance  for  reporting  and  analyzing  data  on  system  failure. 

Interim  Defense  Acquisition  Guidebook.  The  Interim  Defense  Acquisition 
Guidebook  states  that  effective  sustainment  of  weapon  systems  begins  with  the 
design  and  development  of  reliable  and  maintainable  systems  through  the 
continuous  application  of  a  robust  systems  engineering  methodology.  The 
Guidebook  also  states  that  program  managers  shall  conduct  supportability 
analyses  to  achieve  the  most  cost-effective  support  strategy  and  to  foim  the  basis 
for  logistics  support  planning. 

DoD  Manual  4245.7-M.  DoD  Manual  4245. 7-M  states  that  all  system  failures 
should  be  analyzed  in  sufficient  depth  to  identify  the  cause  of  failure  and  the 
necessary  corrective  actions;  that  criticality  of  failures  should  be  prioritized  in 
accordance  with  their  individual  impact  on  operational  performance;  and,  that  a 
central  technical  organization  should  be  responsible  for  implementing  and 
monitoring  the  failure  reporting  system. 


ICAP  III  System  Engineering  Management  Plan 


To  implement  the  DoD  guidance,  in  February  1998,  the  Program  Manager  for  the 
EA-6B  aircraft  directed  Northrop  Grumman  to  develop  and  use  a  Systems 
Engineering  Management  Plan  for  the  ICAP  III  Program  reliability, 
maintainability,  and  failure  analysis.  In  September  1998,  the  contractor 
completed  the  ICAP  III  Systems  Engineering  Management  Plan,  which  states  that 
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reliability  predictions  shall  be  calculated  based  on  anticipated  system  failure  rates. 
The  plan  indicates  that  predictions  calculated  in  the  failure  modes  and  effects  and 
criticality  analysis  document  shall  be  updated  when  a  review  of  the  failures 
identified  during  testing  had  been  analyzed. 


Contract  System  Reliability  Requirements 

In  March  1998,  the  Program  Manager  awarded  a  contract  to  Northrop  Grumman 
for  the  design  and  development  of  the  ICAP  III.  In  the  contract,  the  Program 
Manager  required  Northrop  Grumman  to  ensure  that  system  failure  reporting 
commenced  in  sufficient  time  to  provide  failure-reporting  data  to  support  the 
operational  assessment  and  the  dedicated  developmental  and  operational  tests. 
Additionally,  the  Program  Manager  required  Northrop  Grumman  to  establish  a 
logbook  for  each  ICAP  III  subsystem  to  record  the  failures,  analysis,  and 
corrective  actions  taken.  Further,  to  better  identify  the  sustainment  requirements, 
the  Program  Manager  required  the  contractor  to  perform  system  failure  analysis 
and  subsequently  report  system  reliability  predictions  in  the  system  failure  modes 
and  effects  and  criticality  analysis  document. 


Subcontractor  Submission  of  Failure  Analysis  Data 


The  Program  Manager  did  not  require  the  subcontractor  to  submit,  through  the 
prime  contractor  for  the  ICAP  III  Tactical  Jamming  System  Receiver,  updated 
reliability  prediction  data  needed  to  perform  a  cost-benefit  analysis  study  to 
determine  whether  it  was  more  effective  for  the  Navy  or  for  the  subcontractor  to 
establish  the  facilities  to  provide  supply  support  and  to  repair  the  receiver  system. 

The  subcontractor  for  the  ICAP  III  Tactical  Jamming  System  Receiver  did 
maintain  a  system  failure  logbook  for  developmental  systems  one,  two,  and  three. 
As  of  November  2003,  the  logbook  contained  846  entries;  55  failures  were  not 
closed  and  791  failures  were  closed.  In  maintaining  the  logbook  entries, 
however,  the  subcontractor  did  not  fully  comply  with  requirements  in  the 
development  contract.  Specifically,  the  subcontractor  did  not  always  prioritize 
the  criticality  of  the  failures  or  document  what  corrective  actions  were  taken  to 
close  the  failures  in  the  logbook.  Further,  the  subcontractor  did  not  retain  detailed 
documentation  supporting  the  reason  for  closure  or  nonclosure  of  listed  failures. 
The  subcontractor  stated  that  when  corrective  action  was  needed,  engineering 
changes  were  made  to  the  design  and  hardware,  but  corrective  actions  taken  on 
the  failures  were  not  documented  in  the  logbook.  The  subcontractor  also  stated 
that  an  integrated  process  team  of  program  office  and  contractor  staff  reviewed 
the  failure  reports  and  informally  decided  on  the  appropriate  corrective  actions. 

In  June  2001,  the  contractor  issued  the  final  failure  modes  and  effects  and 
criticality  analysis  document  that  reported  the  subcontractor’s  efforts  to  document 
the  263  system  failures.  Subsequently,  the  subcontractor  continued  to  track, 
analyze,  and  report  on  failures  identified  on  the  three  receivers’  developmental 
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systems.  Without  direction  from  the  EA-6B  Program  Office,  through  the  prune 
contractor,  the  subcontractor  did  not  update  the  reliability  prediction  data  for  the 
Tactical  Jamming  System  Receiver  after  June  2001. 

Cost-Benefit  Analysis  for  the  ICAP  III  Program.  As  a  part  of  the  future 
acquisition  support  strategy  for  the  ICAP  III,  the  Program  Manager  is  considering 
whether  it  is  more  cost-effective  to  use  the  performance-based  logistics  support 
services  offered  by  Northrop  Grumman  or  the  existing  Navy  supply  and 
maintenance  facilities.  The  Interim  Defense  Acquisition  Guidebook  emphasizes 
that  program  managers  should  conduct  supportability  analyses  to  determine  the 
most  cost-effective  support  strategy.  In  the  ICAP  III  Systems  Engineering 
Management  Plan,  the  Program  Manager  directed  Northrop  Grumman  to  use  the 
failure  data  collected  during  development  testing  to  prepare  the  failure  analysis 
and  reliability  prediction  documentation  that  could  be  used  to  complete  a  cost- 
benefit  analysis.  Because  the  Program  Manager  did  not  require  the  contractor  and 
the  Tactical  Jamming  System  Receiver  subcontractor  to  update  the  documentation 
based  on  completed  failure  data,  an  up-to-date,  cost-benefit  analysis  was  not 
available  to  determine  the  most  cost-effective  acquisition  support  strategy  for  the 
Tactical  Jamming  System  Receiver. 


Conclusion 


As  a  result  of  not  being  able  to  accurately  predict  the  expected  reliability  of  the 
Tactical  Jamming  System  Receiver,  the  Program  Manager  may  incur  higher  than 
expected  costs  to  maintain  and  logistically  support  the  receiver  when  the  ICAP  III 
Program  becomes  operational.  Updating  reliability  prediction  data  on  the 
Tactical  Jamming  System  Receiver  would  enable  the  Program  Manager  to  better 
determine  whether  it  is  more  cost-effective  to  use  the  subcontractor  for 
performance -based  logistics  support  or  to  use  existing  Navy  supply  and 
maintenance  facilities  to  provide  supply  support  and  to  repair  the  receiver  system. 
In  addition,  updating  the  reliability  prediction  data  would  enable  the  Program 
Manager  to  better  estimate  the  life-cycle  costs  for  the  ICAP  III  Program. 


Management  Comments  on  the  Finding  and  Audit  Responses 


Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
responding  for  the  Program  Manager  for  the  EA-6B,  nonconcurred  with  the 
finding,  stating  that  the  contractor’s  reliability  and  maintainability  program 
followed  during  the  system  development  phase  of  the  acquisition  process  formed 
the  basis  for  follow-on  phases  of  the  program  and  was  established  to  identify 
failures  during  the  earliest  phase  in  the  design  and  to  implement  the  corrective 
action  at  the  earliest  possible  time  and  at  the  least  cost  to  the  program.  He  stated 
that  the  Program  Manager  actively  pursued  a  policy  that  ensured  that  the 
supportability  and  testability  of  all  ICAP  III  components  would  be  traceable  to 
failure  and  damage  modes  as  predicted,  and  that  every  corrective  action  would  be 
accurately  documented  through  feedback  to  the  design  agents.  The  Deputy 
Assistant  Secretary  also  stated  that  the  data  requested  by  the  audit  team  covered  a 
time  period  of  more  than  2  years  and  needed  to  be  formatted  into  a  useable  report. 
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He  stated  that,  by  mutual  consent,  the  audit  team  and  the  ICAP  III  Integrated 
Process  Team  decided  against  producing  such  a  report  because  it  would  be 
outside  of  the  scope  of  existing  documentation.  The  Deputy  Assistant  Secretary 
stated  that  the  Reliability  Engineer  met  with  the  audit  team  in  May  2004  and 
provided  examples  from  the  Northrop  Grumman  failure  reporting  and  corrective 
action  system  database,  the  reliability  maintainability  and  review  board  database, 
and  meeting  minutes. 

Audit  Response.  The  Program  Manager  did  establish  a  reliability  and 
maintainability  program  to  identify  failures  during  the  earliest  phase  in  the  design 
process  and  to  implement  the  corrective  action  as  early  as  possible.  However,  our 
review  of  failure  logbook  documentation  for  the  Tactical  Jamming  System 
Receiver  indicated  that  the  subcontractor  did  not  execute  the  reliability  and 
maintainability  program  as  required  in  the  development  contract  statement  of 
work  and  the  ICAP  III  Systems  Engineering  Management  Plan.  As  stated  in  the 
draft  report,  we  attempted  to  review  a  judgment  sample  of  the  failure  anomalies 
listed  in  the  receiver  subcontractor  failure  logbook  at  the  subcontractor  facility 
and  were  unable  to  trace  the  line  items  to  a  documented  analysis  to  show 
subcontractor  actions  taken  to  close  each  failure  entry. 


Recommendations,  Management  Comments,  and  Audit 
Response 

C.  We  recommend  that  the  Program  Manager  for  the  EA-6B  Program: 

1.  Require  Northrop  Grumman  Electronic  Systems,  Baltimore, 
Maryland,  through  the  prime  contractor,  to  update  the  failure  report 
logbook  to  prioritize  open  system  failures  for  the  Tactical  Jamming  System 
Receiver  and  to  cross-reference  corrective  actions  taken  to  close  each  failure 
entry. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs), 
responding  for  the  Program  Manager  for  the  EA-6B,  nonconcurred,  stating  that 
the  failure  reporting  and  corrective  action  system  provides  the  means  to  document 
and  improve  the  reliability  of  the  design  through  failure  recording,  analysis,  and, 
if  applicable,  corrective  design  actions.  He  also  stated  that  the  failure  reporting 
and  the  corrective  action  system  reporting  do  not  begin  until  the  system  reaches 
design  maturity,  and  that  the  Reliability  and  Maintainability  Integrated  Process 
Team  deteimined  that  the  failure  reporting  and  corrective  action  system  reporting 
would  begin  when  the  ICAP  III  system  was  sufficiently  mature.  The  Deputy 
Assistant  Secretary  further  stated  that  logbook  entry  anomalies  may  or  may  not  be 
in  the  failure  reporting  and  corrective  action  system  database,  depending  on  the 
development  phase  in  which  the  anomalies  occurred.  Additionally,  he  stated  that 
the  logbooks  were  an  internal  Northrop  Grumman  practice  and  not  a  contract 
requirement.  The  Deputy  Assistant  Secretary  stated  that  the  purpose  of  the  failure 
logbook  was  to  record  anomalies  that  occurred  during  each  shift,  but  that  all 
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anomalies  were  not  recorded  in  the  failure  reporting  and  corrective  action  system 
database.  Finally,  he  stated  that  if  the  anomalies  were  not  placed  into  the  failure 
database,  they  were  described  as  development  notes. 

Audit  Response.  The  Deputy  Assistant  Secretary’s  comments  were  not 
responsive.  Furthermore,  the  Deputy  Assistant  Secretary’s  comment  on  the  start 
of  failure  reporting  is  not  consistent  with  provisions  in  DoD  Manual  4245.7 
which  recommends  that  all  system  failures  be  analyzed,  prioritized,  and 
monitored.  In  this  regal’d,  the  contract  statement  of  work  did  require  Northrop 
Grumman  to  perform  analysis  on  its  failures,  in  a  logbook,  in  sufficient  time  to 
support  the  operational  assessment  and  the  dedicated  development  and 
operational  tests.  As  stated  in  the  draft  report,  the  subcontractor  did  not  retain 
documentation  supporting  the  reason  for  closure  or  nonclosure  of  listed  failures  in 
the  failure  report  logbook  as  required.  Accordingly,  we  request  that  the  Deputy 
Assistant  Secretary  reconsider  his  position  on  the  recommendation  to  require 
Northrop  Grumman  Electronic  Systems,  Baltimore,  Maryland,  through  the  prime 
contractor,  to  update  the  failure  report  logbook  to  prioritize  open  system  failures 
for  the  Tactical  Jamming  System  Receiver  and  to  cross-reference  corrective 
actions  taken  to  close  each  failure  entry. 

2.  Obtain  from  Northrop  Grumman  Electronic  Systems, 
Baltimore,  Maryland,  through  the  prime  contractor,  an  updated  failure 
modes  and  effects  and  criticality  analysis  document  that  includes  an  analysis 
of  corrective  actions  taken  to  resolve  system  failures  that  occurred  after  June 
2001. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
nonconcured,  stating  that  the  Reliability  Prediction  and  the  FMECA  (Failure 
Mode  Effects  and  Criticality  Analysis)  are  tools  used  to  assess  the  proposed 
design  for  compliance  with  requirements,  supportability,  and  testability.  He  also 
stated  that  the  predictions  are  design  assessment  tools  used  in  the  absence  of 
actual  failure  data,  and  that  once  the  design  is  completed  and  solidified,  there  is 
little  payoff  in  performing  updates  unless  major  design  changes  occur.  The 
Deputy  Assistant  Secretary  further  stated  that  once  actual  failure  data  have  been 
obtained,  the  failure  predictions  become  obsolete  and  the  FMECA  should  not  be 
updated. 

Audit  Response.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
comments  were  not  responsive.  As  stated  in  the  draft  report,  the  ICAP  III  System 
Engineering  Management  Plan  addressed  the  need  for  reliability  predictions  and 
the  failure  modes  and  effects  and  criticality  analysis  document.  The  plan  required 
that  reliability  predictions  be  calculated  based  on  anticipated  ICAP  III  system 
failures  and  that  assumptions  made  in  the  ICAP  III  failure  modes  and  effects  and 
criticality  analysis  document  be  updated  when  a  review  of  the  failures  identified 
during  testing  had  been  analyzed.  Because  the  design  for  the  EA-6B  ICAP  III 
Program  has  not  stabilized,  updating  the  FMECA,  as  called  for  in  the  System 
Engineering  Management  Plan,  would  provide  the  Program  Manager  for  the 
EA-6B  with  the  information  he  needs  to  determine  whether  it  is  more  cost- 
effective  to  use  the  subcontractor’s  logistics  support  or  to  use  existing  Navy 
supply  and  maintenance  facilities  to  provide  support  and  to  repair  the  receiver 
system.  Accordingly,  we  request  that  the  Deputy  Assistant  Secretary  reconsider 
his  position  on  the  recommendation  to  obtain  from  Northrop  Grumman  Electronic 
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Systems,  Baltimore,  Maryland,  through  the  prime  contractor,  an  updated  failure 
modes  and  effects  and  criticality  analysis  document  that  includes  an  analysis  of 
corrective  actions  taken  to  resolve  system  failures  that  occurred  after  June  2001. 

3.  Perform  a  cost-benefit  analysis  after  implementing 
Recommendations  C.l.  and  C.2.  to  determine  the  most  cost-effective 
acquisition  support  strategy  for  the  Tactical  Jamming  System  Receiver. 

Navy  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
nonconcurred,  stating  that  the  business  case  analysis  to  determine  the  acquisition 
support  strategy  for  the  Tactical  Jamming  System  Receiver  was  being  performed 
by  the  Navy  Inventory  Control  Point,  which  is  an  independent  program  office  that 
is  not  under  the  control  of  the  Program  Manager  for  the  EA-6B.  He  also  stated 
the  ICAP  III  Integrated  Logistic  Support  Integrated  Process  Team  provided  the 
Navy  Inventory  Control  Point  with  the  latest  reliability  predictions  for  use  in  the 
business  case  analysis  and  that  the  team  was  revising  those  predictions  based  on 
the  results  from  the  dedicated  technical  and  operational  testing  evaluations.  The 
Deputy  Assistant  Secretary  also  stated  that  because  the  test  data  sample  size  was 
not  yet  significant  enough  to  use  in  place  of  the  latest  reliability  predictions,  the 
predicted  reliability  would  be  used  to  forecast  the  support  requirements  for  the 
Tactical  Jamming  System  Receiver. 

Audit  Response.  The  Deputy  Assistant  Secretary’s  comments  were  responsive  to 
the  intent  of  the  recommendation.  A  better  prediction  of  the  reliability  of  the 
Tactical  Jamming  System  Receiver  should  result  from  the  ICAP  III  Integrated 
Logistics  Support  Process  Team’s  revising  earlier  reliability  predictions  for  the 
Tactical  Jamming  System  Receiver  with  reliability  data  from  the  dedicated 
technical  and  operational  testing  to  achieve  results  similar  to  the  action 
recommended. 
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Appendix  A.  Scope  and  Methodology 

We  evaluated  whether  the  EA-6B  Program  Manager  was  cost-effectively 
readying  the  ICAP  III  Program  for  the  production  phase  of  the  acquisition 
process.  Consequently,  we  focused  the  review  on  the  areas  of  test  and  evaluation, 
information  systems  security,  and  system  failure  reporting.  We  performed  this 
audit  from  September  2003  through  June  2004  in  accordance  with  generally 
accepted  government  auditing  standards. 

We  reviewed  documentation  dated  from  July  1997  through  December  2003, 
which  we  obtained  from  the  Naval  Air  Systems  Command,  Program  Management 
Activity-234,  Patuxent  River,  Maryland;  Northrop  Grumman  Integrated  Systems, 
Bethpage,  New  York;  Commander,  Operational  Test  and  Evaluation  Force, 
Norfolk,  Virginia;  Naval  Air  Warfare  Center  Weapons  Division,  Point  Mugu, 
California;  and  Northrop  Grumman  Electronic  Systems,  Baltimore,  Maryland. 

To  accomplish  the  audit  objectives,  we  took  the  following  steps: 

•  We  reviewed  DoD  Instruction  5000.2,  “Operation  of  the  Defense 
Acquisition  System,”  May  12,  2003,  and  Secretary  of  the  Navy 
Instruction  5420. 188E,  “Acquisition  Category  Program  Decision  Process,” 
December  11,  1997,  to  determine  whether  the  EA-6B  Program  Manager 
had  provided  complete  information  as  required  by  DoD  and  Navy  policy 
to  the  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  on  the  operational  assessment  for  the  ICAP  III  Program 
prepared  by  the  Commander,  Operational  Test  and  Evaluation  Force.  To 
make  the  determination,  we  reviewed  the  operational  assessment  and 
interviewed  COMOPTEVFOR  representatives  regarding  the  operational 
assessment  test  results.  We  also  reviewed  the  Navy  program  decision 
briefing  used  by  the  milestone  decision  authority  to  make  the  low-rate 
initial  production  decision. 

•  We  reviewed  DoD  Instructions  5200.40,  “Department  of  Defense 
Information  Technology  Security  Certification  and  Accreditation 
Process,”  December  30,  1997,  and  DoD  Manual  8510.1-M,  “Department 
of  Defense  Information  Technology  and  Security  Certification  and 
Accreditation  Process  Application  Manual,”  July  2000,  to  determine 
whether  the  Certification  Authority  ensured  that  system  security 
verification  tasks  as  required  by  DoD  policy  had  been  completed  for  the 
ICAP  III  Program  and  whether  the  Designated  Approving  Authority 
ensured  that  the  required  validation  tasks  were  completed  before  issuance 
of  the  IATO.  In  addition,  we  determined  whether  the  Program  Manager 
had  fully  completed  the  DITSCAP  to  provide  decision  makers  with  full 
assurance  that  the  ICAP  III  information  systems  will  function  correctly 
and  meet  performance  requirements. 

•  We  reviewed  the  Interim  Defense  Acquisition  Guidebook  and  DoD 
Manual  4245. 7-M,  “Transition  from  Development  to  Production,” 
September  1985,  to  determine  whether  the  Program  Manager  followed 
established  policies  and  guidance  when  he  provided  Northrop  Grumman 
direction  to  perform  reliability  and  maintainability  failure  analysis  for  the 
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Tactical  Jamming  System  Receiver.  We  reviewed  and  analyzed  the 
composite  list  of  open  Tactical  Jamming  System  Receiver  trouble  reports, 
the  Systems  Engineering  Management  Plan,  the  failure  modes  and  effects 
and  criticality  analysis  document,  and  subcontractor  submission  of  failure 
analysis  data  in  the  failure  report  logbook  to  determine  corrective  actions 
taken  to  reduce  future  hardware  failures.  In  addition,  we  interviewed 
representatives  from  Northrop  Grumman  Integrated  Systems  and  Northrop 
Grumman  Electronic  Systems. 

During  the  review,  the  Program  Manager’s  representatives  did  not  respond  timely 
to  the  audit  team’s  repeated  requests  for  documentation  to  support  the  life-cycle 
cost  estimate  that  the  Program  Manager  reported  in  the  October  2003  acquisition 
program  baseline  agreement.  As  a  result,  the  audit  team  prepared  a  potential 
finding  with  recommendations  to  ensure  that  the  Program  Manager  documented 
his  future  life-cycle  cost  estimates.  The  audit  team  presented  this  potential  issue 
in  a  discussion  draft  report  to  the  Program  Manager’s  representatives  in 
March  2004.  In  May  2004,  the  representatives  presented  the  audit  team  with  the 
documentation  that  supported  the  October  2003  acquisition  program  baseline 
agreement,  hi  future  audit  reviews,  the  Program  Manager  should  ensure  that  his 
representatives  respond  more  timely  to  audit  requests  for  documentation. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Use  of  Technical  Assistance.  A  computer  engineer  from  the  Technical 
Assessment  Division,  Office  of  the  Assistant  Inspector  General  for  Auditing  of 
the  Department  of  Defense  participated  in  the  review  of  the  ICAP  III  Program. 
Specifically,  the  computer  engineer  evaluated  the  information  assurance  process 
and  the  level  of  software  failures  that  required  alternative  system  solutions. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  DoD.  This  report  provides  coverage  of 
the  DoD  Weapons  System  Acquisition  high-risk  area. 


Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26,  1996, 
and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures,” 
August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provide  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  In  accordance 
with  DoD  policy,  acquisition  managers  are  to  use  program  cost,  schedule,  and 
performance  parameters  as  control  objectives  to  implement  the  requirements  of 
DoD  Directive  5010.38.  Accordingly,  we  limited  our  review  to  management 
controls  directly  related  to  areas  of  test  and  evaluation,  information  assurance, 
and  design  failure  reviews  for  the  ICAP  III  Program.  We  also  assessed 
management’s  self-evaluation  of  those  controls. 
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Adequacy  of  Management  Controls.  We  identified  management  control 
weaknesses,  as  defined  by  DoD  Instruction  5010.40,  relating  to  reporting  of  test 
and  evaluation  results,  information  assurance,  and  system  failure  reviews. 
Specifically,  the  Program  Manager  did  not  provide  the  COMOPTEVFOR 
operational  assessment  to  the  milestone  decision  authority  as  required  for  the 
Navy  Program  Decision  Meeting  or  fully  verify  that  analysis  and  corrective 
action  had  been  performed  on  hardware  failures.  In  addition,  the  Designated 
Approving  Authority  for  the  Naval  Air  Systems  Command  did  not  follow 
procedures  for  issuing  the  Program  Manager  an  IATO.  Recommendations  A.I., 

A. 2.,  B.I.,  B.2.,  C.I.,  and  C.2.,  if  implemented,  will  improve  the  overall 
management  of  the  ICAP  III  Program.  A  copy  of  this  report  will  be  provided  to 
the  senior  official  responsible  for  management  controls  in  the  Department  of  the 
Navy. 

Adequacy  of  Management’s  Self-Evaluation.  The  Program  Executive  Officer 
for  Tactical  Aircraft  Programs  performed  annual  reviews  of  the  ICAP  III 
Program’s  assessable  units  to  satisfy  the  management  control  requirements.  The 
Program  Executive  Officer  used  executive  acquisition  review  boards,  program 
management  reviews,  readiness  reviews,  quality  assessments,  financial  system 
reviews,  audits,  independent  evaluations,  inspections,  internal  reviews, 
investigations  and  consulting  reviews  to  evaluate  the  assessable  units  for  the 
program.  The  Program  Executive  Officer  based  his  annual  statement  of  assurance 
on  results  noted  during  the  reviews  of  the  assessable  units.  However,  in  the 
self-evaluations,  the  Program  Executive  Officer  did  not  identify  the  specific 
management  control  weaknesses  because  the  self-evaluations  did  not  review  those 
specific  areas  as  pail  of  the  assessable  units. 


Prior  Coverage 


During  the  last  5  years,  the  General  Accountability  Office  (GAO)  has  issued  four 
reports  that  discuss  the  EA-6B  Prowler  aircraft  and  its  role  in  airborne  electronic 
attack  needed  for  suppression  of  enemy  air  defenses.  Unrestricted  GAO  reports 
can  be  accessed  over  the  Internet  at  http://www.gao.gov. 


GAO 


GAO  Report  No.  GAO-04-112,  “Military  Readiness:  DoD  Needs  to  Reassess 
Program  Strategy,  Funding  Priorities,  and  Risks  for  Selected  Equipment,” 
December  19,  2003 

GAO  Report  No.  GAO-03-51,  “Electronic  Warfare:  Comprehensive  Strategy 
Still  Needed  for  Suppressing  Enemy  Air  Defenses,”  November  25,  2002 

GAO  Report  No.  GAO-Ol-28,  “Electronic  Warfare:  Comprehensive  Strategy 
Needed  for  Suppressing  Enemy  Air  Defenses,”  January  3,  2001 

GAO  Report  No.  GAO-00- 164,  “Contingency  Operations:  Providing  Critical 
Capabilities  Poses  Challenges,”  July  6,  2000 
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Appendix  B.  Other  Matters  of  Interest 


During  the  audit,  we  noted  a  matter  of  interest  concerning  test  deficiencies  in  the 
ICAP  III  software. 


Test  Deficiencies  in  ICAP  III  Software 

In  March  2003,  the  Program  Manager  released  build  three  of  the  ICAP  III 
software  to  the  Naval  Air  Systems  Command  Technical  Assurance  Board  for  an 
independent  technical  evaluation.  Concurrently,  he  completed  the  development 
of  build  four  and  readied  it  for  formal  qualification  testing.  During  the  integration 
test  processes  for  the  two  builds,  the  program  software  team  identified  303 
priority  three  software  deficiencies.  As  defined  in  Military  Standard  498, 
“Software  Development  and  Documentation,”  December  5,  1994,  priority  three 
software  deficiencies  are  rated  as  manageable,  provided  that  the  software 
development  team  can  identify  alternative  solutions. 

Through  the  Software  Development  Plan,  the  Program  Manager  established  a 
joint  Government  and  contractor  software  integrated  process  team  to  manage  the 
integrated  software  testing  at  the  Naval  Air  Warfare  Center  Weapons  Division, 
Point  Mugu,  California.  The  EA-6B  Program  Manager  is  relying  on  the 
program’s  software  integrated  process  team  to  monitor  ICAP  III  software 
development  testing  and  to  track  and  resolve  deficiencies  noted  with  builds  three 
and  four.  In  November  2003,  the  Government  software  integrated  process  team 
leader  stated  that  an  analysis  of  the  303  software  deficiencies  should  be 
completed  to  deteimine  the  effect  on  computer  software  configuration  items. 

Also,  in  November  2003,  a  representative  for  COMOPTEVFOR  stated,  when 
asked  about  the  number  of  software  deficiencies  for  the  tactical  display,  that  if  the 
alternative  solutions  resulted  in  more  display  screens  than  originally  planned,  an 
aircraft  operator  could  not  effectively  perform  the  airborne  electronic  attack 
mission.  As  of  June  2004,  the  integrated  process  team  had  not  analyzed  the  level 
of  risk  associated  with  the  303  priority  three  software  deficiencies. 

By  not  analyzing  the  level  of  risk  associated  with  the  number  of  priority  three 
deficiencies,  the  integrated  process  team  did  not  fully  execute  the  software 
development  processes  that  the  Program  Manager  planned.  Through  the 
performance  of  a  detailed  risk  analysis,  the  Program  Manager  could  better  inform 
the  milestone  decision  authority  of  the  effects  of  the  alternative  solutions  on  the 
performance  requirements  for  the  ICAP  III  Program  software. 
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Appendix  C.  Upgrades  for  the  Improved 

Capability  III  System 


The  ICAP  III  Program  is  one  of  several  planned  improvements  to  the  EA-6B 
Prowler  aircraft  that  will  ensure  that  the  aircraft  remains  the  world’s  premier 
tactical  electronic  warfare  platform  and  is  able  to  sustain  the  DoD  airborne 
electronic  attack  mission  through  2015. 

The  ICAP  III  Program  is  a  Navy  Acquisition  Category  II  program  that  will 
provide  the  electronic  attack  mission  with  selective-reactive  jamming  capability 
in  a  wider  frequency  range,  improve  information  display  and  battle  management 
capability,  accommodate  provisions  for  a  modern  data  link,  and  improve  the 
reliability  of  affected  systems  through  the  upgrades  to  the  following  systems: 

•  New  On-board  Receiver  Subsystem.  The  Tactical  Jamming  System 
Receiver  system  replaces  the  ALQ-99  on-board  weapon  system.  The 
receiver  gives  the  EA-6B  the  ability  to  employ  selective-reactive  jamming 
of  enemy  integrated  air  defense  systems.  This  upgrade  will  recover  the 
effectiveness  lost  to  modern  electronic  warfare,  acquisition  radars,  and 
surface  to  air  missiles  by  applying  intense,  narrow- spot  jamming.  The 
new  receiver  group  has  10  weapons-replaceable  assembly  units. 

•  Tactical  Display  Subsystem.  The  new  Tactical  Display  Subsystem, 
combined  with  an  Interface  Unit,  will  provide  display  and  operator 
interface  improvements  and  eliminate  the  use  of  the  laptop  computer  in 
the  cockpit.  This  technology  will  result  in  displays  that  allow  the  aircrews 
to  focus  on  the  relevant  elements  of  friendly  forces,  the  enemy,  allied 
forces,  weather,  terrain,  and  numerous  other  fixed  and  dynamic  data  that 
comprise  the  “tactical  picture.” 

•  Provisions  for  Connectivity  Integration.  To  function  as  the  sole 
airborne  tactical  jammer  for  DoD,  the  EA-6B  aircraft,  inherently,  must  be 
part  of  the  larger  “Sensor  to  Shooter”  architecture.  As  such,  the  EA-6B 
aircraft  must  be  capable  of  sending,  receiving,  and  coordinating  accurate 
and  timely  information  from  other  air,  land,  and  sea  sensors  and  command 
and  control  centers.  The  addition  of  the  Multifunctional  Information 
Distribution  System  Link-16  network  will  provide  this  capability  to  the 
EA-6B  aircraft. 
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Appendix  D.  Additional  Recommendations  for 

the  Operational  Assessment  of  the 
Improved  Capability  III  System 


The  Commander,  Operational  Test  and  Evaluation  Force  issued  an  operational 
assessment  for  the  EA-6B  ICAP  III  weapon  system  on  June  5,  2003.  The 
COMOPTEVFOR  identified  50  deficiencies  of  the  EA-6B  ICAP  III,  which  were 
not  critical  to  mission  accomplishment,  but  detracted  from  the  overall  EA-6B 
operator  efficiency  or  operator  situational  awareness.  The  COMOPTEVFOR 
suggested  that  the  50  deficiencies  be  corrected  and  verified  during  the  dedicated 
operational  test  and  evaluation  phase.  The  50  additional  deficiencies  identified 
for  the  ICAP  III  system  include: 

1.  hivestigate  and  correct  the  cause  of  erroneous  misidentification  of  threat 
emitters. 

2.  Resolve  deficiencies  of  a  single  strike  group  route  when  implementing 
reactive  assignments. 

3.  Investigate  and  correct  the  Tactical  Jamming  System  Receiver  lockups  that 
occurred  during  various  mission  phases  throughout  operational  assessment 
testing. 

4.  Investigate  and  correct  the  areas  of  risk  that  cause  an  excessive  heads-down 
time  by  the  pilot. 

5.  Correct  the  inability  to  choose  between  a  moving  map  and  a  static  display. 

6.  Correct  the  illogical  color  display  of  emitters. 

7.  Investigate  and  correct  the  unreliable  nature  of  built-in  tests. 

8.  Investigate  and  correct  inconsistent  Reactive  Assignment  initiation  time. 

9.  Investigate  and  correct  the  area  of  risk  that  prevents  the  operator  from 
changing  the  classification  of  a  complex  ambiguous  emitter. 

10.  Investigate  and  correct  the  inconsistent  geo-locations. 

11.  Investigate  and  correct  the  inadequacies  of  the  Tactical  EA-6B  Mission 
Planning  System  mission  planning  tools. 

12.  Investigate  and  correct  the  inadequacies  of  the  Tactical  EA-6B  Mission 
Planning  System  user  interface. 

13.  Investigate  and  correct  the  inadequacies  of  the  Tactical  EA-6B  Mission 
Planning  System  emitter  libraries. 
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14.  Investigate  and  correct  the  cumbersome  strike  group  route  adjustment. 

15.  Correct  the  Operational  Navigational  Chart  and  the  background  images  in  the 
Tactical  Pilotage  Chart  so  that  overlays  are  not  washed  out. 

16.  Investigate  and  correct  the  excessive  generation  of  reactive  assignments. 

17.  Investigate  and  correct  the  cumbersome  display  of  new  emitter  information  on 
the  Alarms  Page. 

18.  Improve  the  speed  of  Electronic  Counter  Measure  Officer  2  and  3’s  cursor. 

19.  Improve  the  functionality  of  Tactical  Display  Subsystem  cursors. 

20.  Correct  the  obstruction  of  the  upper  10  percent  of  the  pilot’s  display. 

21.  Improve  display  startup  defaults  on  the  geo-location  page. 

22.  Investigate  and  correct  the  area  of  risk  that  permits  the  pilot  to  inadvertently 
turn  off  the  anti-collision  light. 

23.  Correct  Pocket  Checklist  areas  of  risk. 

24.  Correct  Naval  Air  Training  and  Operating  Procedures  Standardization  areas 
of  risk. 

25.  Investigate  and  correct  the  nonoptimized  jammer  assignment  logic  for  band  9 
and  band  10  Reactive  Assignments. 

26.  Incorporate  A-F  functionality. 

27.  Correct  cumbersome  de-cluttering  functionality. 

28.  Improve  Bulls  eye  adjustment  procedures. 

29.  Investigate  and  correct  the  area  of  risk  that  caused  emitter  symbols  to  be 
displayed  without  associated  parametric  information. 

30.  Incorporate  a  radar  cursor  slew  function. 

31.  Improve  mission  clock  functionality. 

32.  Improve  soft  keypad  data  entry  procedures. 

33.  Improve  training  for  ICAP  III  version  7/8  of  the  Tactical  EA-6B  Mission 
Planning  System. 

34.  Incorporate  a  frequency  management  planning  tool  in  version  7/8  of  the 
Tactical  EA-6B  Mission  Planning  System  that  assists  the  operator  with 
identifying  and  correcting  frequency  conflicts,  such  as  Joint  Restricted 
Frequency  List. 
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35.  Investigate  the  area  of  risk  that  prevented  operators  from  making  Alarm 
Assignments. 

36.  Investigate  and  correct  the  area  of  risk  that  allows  operators  to  inadvertently 
purge  the  active  emitter  file. 

37.  Investigate  and  correct  the  area  of  risk  that  caused  the  Tactical  Display 
Subsystem  to  revert  to  status  page  upon  library  loads. 

38.  Incorporate  a  transmitter  centroiding  indication  on  the  jammer  assignment 
status  display. 

39.  Investigate  and  correct  the  area  of  risk  that  caused  an  inconsistent  display  of 
information  when  changing  the  size  of  the  emitter  ellipse. 

40.  Improve  jammer  pod  power  failure  and  steering  failure  alerts. 

41.  Improve  the  display  of  changes  to  the  jammer  assignment  status  page. 

42.  Improve  the  alarm  zone  frequency  adjustment  sequence. 

43.  Correct  the  font  size  for  easier  reading. 

44.  Improve  the  usability  of  target  tracker  lines. 

45.  Incorporate  a  frequency/azimuth  page  on  the  pilot’s  Tactical  Display 
Subsystem. 

46.  Resolve  the  conflicting  information  presented  to  the  operator  on  the  jammer 
assignment  status  page. 

47.  Correct  the  altered  Library  Summary  Mission  Plan  page  priorities. 

48.  Change  recorder  reproducer  set  fail  light  to  reflect  presence  of  the  data  storage 
memory  unit. 

49.  Incorporate  a  threat  warning  or  radar  homing  and  warning  system. 

50.  Streamline  ICAP  III  training  for  fleet  operator  use. 
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Appendix  E.  Navy  Development  and  Operational 

Test  Procedures 


Through  the  test  and  evaluation  process,  the  Navy  will  evaluate  technical 
performance  and  system  maturity  to  determine  whether  the  ICAP  III  system  is 
operationally  effective,  suitable,  and  survivable  against  identified  threats.  The 
Navy’s  process  includes  developmental  test,  technical  evaluation,  operational  test, 
and  follow-on  test  phases  to  determine  whether  the  ICAP  III  Program  hardware 
and  software  have  been  adequately  readied  for  production.  A  description  of  the 
Navy’s  four  test  phases  for  the  ICAP  III  Program  follows. 


Developmental  Testing 


Developmental  testing  supports  multiple  sub-phases  of  ICAP  III  Program 
contractor  and  Government  testing.  The  objective  of  developmental  testing  is  to 
ensure  that  the  designed  hardware  subsystems  and  related  software  performance 
meet  the  system  specifications.  Additionally,  this  test  phase  is  to  verify  whether 
the  ICAP  III  hardware  and  software  modifications  are  properly  installed,  meet 
safety  of  flight  requirements,  function  correctly,  and  satisfy  top-level  integrated 
system  performance  requirements.  The  Program  Manager  used  the  results  of  this 
phase  to  determine  the  integrated  system’s  potential  to  meet  required  operational 
characteristics  in  support  of  the  low-rate  initial  production  decision  and  to 
determine  the  degree  of  readiness  to  proceed  to  the  technical  evaluation. 

Northrop  Gnrmman  developed  three  complete  sets  of  hardware  subsystems  to  use 
for  developmental  testing.  The  contractor  placed  system  one  at  the  Naval  Air 
Warfare  Center  Weapons  Division,  Point  Mugu,  California;  and  systems  two  and 
three  on  two  EA-6B  test  aircraft.  Northrop  Gnrmman  also  planned  to  develop 
five  increments  of  the  ICAP  III  Program  software.  As  of  June  2004,  the 
contractor  had  developed  four  increments.  The  Program  Manager  used  the 
contractor’s  third  increment  for  the  Navy’s  technical  evaluation  process. 
Operational  testers  are  using  the  fourth  software  increment  in  the  operational  test 
phase. 

Operational  Assessment.  During  the  developmental  test  phase,  the  Navy 
independent  test  agency  performed  an  operational  assessment  to  determine  the 
potential  operational  effectiveness  and  potential  operational  suitability  of  the 
EA-6B  ICAP  III  system.  In  June  2003,  COMOPTEVFOR  completed  the  ICAP 
III  Program  operational  assessment.  The  Navy  used  the  results  of  the  operational 
assessment  to  support  a  decision  to  begin  low-rate  initial  production.  See 
finding  A  for  the  results  of  that  assessment. 
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Technical  Evaluation 


During  the  Navy  technical  evaluation  phase,  the  Naval  Air  Systems  Command’s 
Technical  Assurance  Board  and  the  ICAP  III  Naval  Strike  Air  Test  and 
Evaluation  Squadron  23  performed  independent  testing  of  ICAP  III  hardware  and 
software.  To  perform  the  evaluation,  the  squadron  used  its  own  pilots, 
maintenance  services,  safety  oversight,  and  facility  support,  hi  July  2003,  the 
Naval  Air  Squadron  began  testing  the  ICAP  III  hardware  and  software  that  had 
been  integrated  into  the  two  EA-6B  test  aircraft.  The  Naval  Air  Squadron  used 
the  results  of  this  phase  to  evaluate  the  readiness  of  the  ICAP  III  to  begin  the 
operational  testing  phase.  The  Program  Manager  will  also  use  the  results  of  the 
technical  evaluation  to  support  the  ICAP  III  Program  full-rate  production 
decision. 


Operational  Testing 


The  Navy  operational  test  and  evaluation  phase  will  determine  whether  the 
ICAP  III  system  is  operationally  effective  and  suitable  under  realistic  conditions. 
This  test  phase  also  will  deteimine  whether  critical  operational  issue  deficiencies 
that  were  previously  identified  have  been  resolved  and  assess  the  potential  effect 
that  system  limitations  may  have  on  actual  combat  operations. 

Operational  testing  will  be  conducted  on  production-representative  ICAP  III 
hardware  and  software.  At  the  conclusion  of  the  tests,  the  Commander, 
Operational  Test  and  Evaluation  Force  will  provide  an  independent  and  objective 
evaluation  report  to  the  milestone  decision  authority  stating  whether  the  ICAP  III 
is  operationally  effective  and  suitable.  Successful  accomplishment  of  operational 
evaluation  will  support  a  fleet  introduction  recommendation  and  support  an 
ICAP  III  Program  full-rate  production  decision. 


Follow-on  Test  and  Evaluation 


The  Program  Manager  plans  to  use  a  follow-on  test  and  evaluation  phase  to  test 
additional  planned  EA-6B  upgrades  that  will  not  be  available  when  the  ICAP  III 
system  is  produced  and  installed  on  the  aircraft.  The  systems  that  are  scheduled 
for  follow-on  testing  include  the  Multifunctional  Information  Distribution  System 
Link- 16  network  and  the  Joint  Mission  Planning  System. 
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Appendix  F.  Audit  Response  to  Management 

Comments  on  the  Report 


Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs) 
Comments 


The  Deputy  Assistant  Secretary  of  the  Navy  (Air  Programs),  Office  of  the 
Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition), 
responding  for  the  Program  Manager  for  the  EA-6B,  provided  additional 
comments  on  the  EA-6B  Program  Office’s  cooperation  with  the  audit  team.  The 
complete  text  of  the  management  comments  on  statements  in  the  draft  report  is  in 
the  Management  Comments  section  of  this  report. 

Navy  Comments.  The  Deputy  Assistant  Secretary  stated  that  the  Program 
Manager,  through  the  ICAP  III  Integrated  Process  Team,  had  fully  cooperated 
with  the  audit  team  in  every  Naval  Air  Systems  Command  meeting  and  field  site 
visit. 

Audit  Response.  The  Program  Manager,  through  the  ICAP  III  Integrated  Process 
Team,  generally  complied  with  the  audit  team’s  request  for  meetings  and 
documentation.  However,  as  stated  in  the  draft  audit  report,  we  experienced 
difficulty  in  timely  obtaining  documentation  that  supported  the  ICAP  III  system 
life-cycle  cost  estimate.  We  requested  that  this  information  be  provided  for 
analysis  in  November  2003  at  the  beginning  of  the  audit.  Additionally,  we 
experienced  delays  in  discussing  the  ICAP  III  failure  reporting  and  reliability 
issues  with  the  reliability  and  maintainability  Integrated  Process  Team  Leader. 
Again,  we  requested  to  hold  the  meeting  in  November  2003.  The  Program  Office 
did  not  make  knowledgeable  personnel  available  to  meet  and  discuss  the  cost 
estimating  and  the  failure  reporting  issues  until  after  we  prepared  our  proposed 
draft  report  and  staffed  it  with  the  ICAP  III  Integrated  Process  Team  in  May 
2004.  Earlier  meetings  with  the  knowledgeable  personnel  would  have  facilitated 
the  conduct  of  the  audit  and  reduced  misunderstanding  between  the  auditors  and 
the  Program  Office  staff. 


33 


Appendix  G.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Under  Secretary  of  Defense  (Comptroller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Operational  Test  and  Evaluation 
Director,  Program  Analysis  and  Evaluation 


J  oint  Staff 

Director,  Joint  Staff 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Manpower  and  Reserve  and  Affairs) 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Commander,  Naval  Air  Systems  Command 

Program  Executive  Officer  for  Tactical  Aircraft  Programs 
Program  Manager,  EA-6B  Improved  Capability 
Naval  Inspector  General 
Auditor  General,  Department  of  the  Navy 
Commander,  Operational  Test  and  Evaluation  Force 


Department  of  the  Air  Force 

Auditor  General,  Department  of  the  Air  Force 


Other  Defense  Organization 

Director,  Defense  Contract  Management  Agency 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Aimed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency  and  Financial  Management,  Committee 
on  Government  Reform 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Information  Policy,  Intergovernmental  Relations, 
and  the  Census,  Committee  on  Government  Reform 
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Department  of  the  Navy  Comments 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  ASSISTANT  SECRETARY 
RESEARCH,  DEVELOPMENT  AND  ACQUISITION 
1000  NAVY  PENTAGON 
WASHINGTON  DC  20350-1000 


AUG  0  5  2(304 


MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  ASSISTANT  INSPECTOR 
GENERAL  FOR  AUDITING 

SUBJECT:  DODIG  DRAFT  REPORT:  ACQUISITION  OF  THE  EA-6B  IMPROVED 
CAPABILITY  III  PROGRAM  (PROJECT  NO.  D2003AE-0190) 

We  have  reviewed  your  draft  report  dated  June  1 8,  2004.  Our  detailed  comments  on  the 
findings  and  the  recommendations  are  attached. 


William  M.  Balderson 

Deputy  Assistant  Secretary  of  the  Navy 

(Air  Programs) 


Attachment: 
As  stated 


cc: 


NAVIG-4 
NAVAIR  (09G) 


37 


DEPARTMENT  OF  THE  NAVY  RESPONSE  TO 
DODIG  DRAFT  REPORT  ON  ACQUISITION  OF  THE 
EA-6B  IMPROVED  CAPABILITY  (ICAP)  HI  PROGRAM 
(PROJECT  NO.  D2003AE-01 90) 

INTRODUCTION 

On  IS  June  200*  the  Department  of  Defense  Inspector  General  (DOD  Kj)  for  Auditing  provided  a  draft 
repon  for  review  and  comment.  The  report  is  the  culmination  of  an  audit  of  the  EA-6B  fCAP  IQ 
Program.  The  audit  commenced  in  September  2003  and  ended  in  June  2004. 

The  DOD  K3  Program  Director's  decision  to  support  a  review  of  the  proposed  draft  report  prior  to  release 
was  beneficial  to  »U  concerned.  The  "read-ahead”  reduced  confusion  and  provided  both  parties  the  time 
to  clari  fy  their  positions  and  assumptions.  This  had  the  positive  effect  of  saving  time  and  consequently 
valuable  program  assets  for  both  parties. 

DOD  1G  AND  PMA-234  COOPERATION 

In  a  Memorandum  dated  18  August  2003,  the  Acquisition  Management  Directorate  initiated  the  audit  of 
the  Acquisition  of  the  EA-6B  ICAP  III  Program.  A  follow-up  memo  dated  25  August  2003  from  the 
Audit  Team  Leader,  requested  an  advanced  delivery  of  ICAP  IQ  Program  documentation.  From  that 
point  in  the  audit  to  the  final  days  just  before  the  receipt  of  the  draft  audit  report  PMA-234  through  the 
ICAP  III  Integrated  Program  Team  <IPT)  has  fully  cooperated  with  the  audit  team  in  support  of  every 
NAVA1R  meeting  and  field  site  visit. 

Enclosure  (1).  DOD  IG  Audit  Documentation  Delivery  Index  lists  the  documentation  and  information 
delivered  to  the  audit  team  over  the  course  of  the  audit..  The  list  is  subdivided  beginning  with  the  initial 
document  request  issued  by  the  25  August  memo  out  to  the  final  site  visit  to  PMA-234  on  3  May  2004. 
The  list  also  indicated  the  mode  of  delivery,  the  date  provided,  the  document  security  clasriftcaiion,  and 
miscellaneous  notes  and  reference  information. 

Enclosure  (2),  EA-6B  ICAP  in  DOD  IG  Audit  Matrix  is  •  cross  reference  to  the  items  in  the  DOD  IG 
Audit  Documentation  Index  to  the  Critical  Program  Management  Elements  (CPMEs)  used  as  a  basis  for 
die  audit  as  indicated  in  the,  "Presentation  to  PMA-234  EA-6B  Program  Office’*,  presented  by  the  DOD 
IG  Team  Leader  to  PMA-234  on  8  September,  2003.  Each  document  provided  to  the  audit  team  was 
mapped  to  the  CPMEs  to  illustrate  the  depth  and  coverage  of  information  provided  in  support  of  the  audit 
teams  own  evaluation  criteria 

RESPONSE  FORMAT 

The  NAVAIR  IG  Audit  Liaison  Officer  provided  the  format  guidance  used  to  respond  to  the  audit 
findings  and  recommendations.  The  alphatounteric  conventions  used  in  each  finding  and 
recommendation  was  followed  in  the  responses  for  continuity. 

In  each  case  the  weeding  in  the  finding  or  recommendation  is  repeated.  The  comments  to  a  finding  begin 
with  either  CONCUR,  PARTIALLY  CONCUR,  OR  DO  NOT  CONCUR  WITH  THIS  FINDING. 
Comments  to  a  recommendation  begin  with  either  CONCUR,  PARTIALLY  CONCUR  OR  DO  NOT 
CONCUR  WITH  THE  RECOMMENDATION. 


jUSEnntopure  1 1  f 
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Respowe  to  DO D IG  Draft  Report  (Pnjccl  No.  D2003AE-GI90) 

PM  A -234  EA-6B ICAP  III  TEAM  RESPONSES  TO  DOD  !G  AUDIT  FINDINGS  AND 
RECOMMENDATIONS 

Finding  A;  Readiness  for  Low<R*te  Initial  Production 

The  Program  Manager  provided  the  Assistant  Secretary  of  the  Navy  (Research*  Development,  and 
Acquisition)  with  incomplete  information  on  the  operational  assessment  of  the  ICAP  HI  Program  that  the 
Commander*  Operational  Test  and  Evaluation  Force  (COMOPTEVFDR)  prepared  in  support  on  the  low- 
rate  initial  prediction  decision.  Specifically,  the  Program  manager  did  not  provide  the  details  for  the 
following  operational  test  results: 

•  The  high  false  emitter  display  ratio1  substantial  ly  reduced  overall  mission  effectiveness  by 
overloading  the  aircrew's  displays  with  erroneous  symbols  and  caused  significant  problems  in 
the  proper  use  of  reactive  assignments. 

•  The  lack  of  a  fully  functional  keypad  prevented  the  teat  aircrew  from  using  the  ICAP  Hi  weapon 
system  effectively. 

•  The  Tactical  EA-6B  Mission  Planning  System  erroneously  displayed  data  (lost  or  frozen  data) 
because  of  frequent  system  lockups,  which  presented  major  obstacles  to  effective  pre flight 
mission  planning  and  post-night  analysts. 

•  Fifty  addition*!  performance  deficiencies  for  the  ICAP  III  systems  required  correction  because 
they  detracted  from  the  overall  EA-6B  efficiency  or  operator  situational  awareness. 

The  Program  Manager  did  not  provide  the  COMOPTEVFOR  (COTF)  operational  assessment  to  the 
Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  as  required  for  Navy  program 
milestone  decision  meetings  because  the  Program  Manager  limited  his  presentation  of  the  COTF  test 
results  to  their  conclusion  that  the  ICAP  III  was  potentially  operationally  effective  and  suitable,  the 
ratings  for  the  critical  operational  issues,  and  a  listing  of  the  50  additional  deficiencies,  The  briefing  did 
not  describe  how  the  deficiencies  affected  operational  effectiveness  and  suitability.  As  a  result,  the 
Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition)  approved  the  Program 
Manager's  request  in  June  2003  to  procure  10  ICAP  HI  systems  for  low-rate  initial  production  and  the 
Navy  increased  the  risk  that  it  will  incur  costly  retrofit  expenses  to  correct  the  design  deficiencies  for 
those  systems  at  the  completion  of  the  dedicated  operational  test  and  evaluation  phase. 

DO  NOT  CONCUR  WITH  THE  FINDING 

1 .  COMOPTEVFOR  (COTF)  is  the  only  releasing  authority  for  the  OA  Report  and  any  operational 
test  report.  The  OA  Report  was  delivered  to  ASN(RD&A)  per  receipt  on  record  at  COTF. 
Enclosure  (3).  The  report  was  received  by  the  ASN(RDA)-fitaff  in  June  2003.  This  fact  was 
provided  to  the  auditors  during  their  visit  to  the  PM  A  an  24-25  March  2004. 

2.  The  Program  Manager  for  ICAP  III  did  present  information  at  the  LR  IP  decision  that  addressed 
details  for  the  operational  test  results,  including  information  regarding  the  nature  of  the 
deficiencies  that  affected  the  effecti  vity  and  suitability  findings  from  OOTF.  A  summary  of  the 
OA  Report  was  provided  on  page  7  of  the  NPDM.  Additionally  back-up  material  was  provided 
that  discussed  the  4  operational  mission  failures  experienced  during  OA  and  the  corrective  action 
plans  with  current  status  (pages  31-35).  Backup  materia)  was  also  provided  that  not  only 
identified  the  high  and  moderate  effectively  and  suitability  risks,  but  also  discussed  status  of 
correction  and  identified  all  minor  COIs  (pages  38-42),  These  back-up  charts  were  discussed  with 
ASNfRDA)  during  the  NPDM.  A  copy  of  the  NPDM  briefing  was  provided  to  the  audit  team  on 
two  separate  occasions,  10  September  2003  and  again  on  14  October  2003. 
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3,  For  informational  purpose*  only,  detailed  correction  status  is  provided  for  the  4  items  specifically 

called  oral  by  DODIG. 

•  “The  high  false  emitter  display  ratio3  substantially  reduced  overall  mission  effectiveness  by 
overloading  the  aircrew's  displays  with  erroneous  symbols  and  caused  significant  problems  in 
the  proper  use  of  reactive  assignments.'' 

e>  As  discussed  in  the  NFDM  briefing,  a  number  of  corrections  were  implemented  id  the 
May  and  June  2003  software  builds  that  addressed  this  issue.  Corrections  made  included 
an  increase  to  the  number  of  cued  receiver  pulses  required  for  a  new  active  emitter  file, 
(he  modification  of  false  report  suppression  logic  to  adjust  RF  match  tolerances, 
correction  of  the  logic  for  merging  and  purging  emitter  reports,  a  reduction  of  tbe  age-out 
times  for  emitters  to  limit  the  creation  of  false  alarms  from  residue,  and  the  addition  of 
swiicbable  attenuation  to  reduce  spurs.  Additionally,  the  1CAP  Ill  Program  bad  adopted 
an  evolutionary  develop  meet  approach  to  address  the  reactive  assignment  (RA) 
capability.  Cased  on  test  data,  RA  improvements  were  achieved  via  ALQ-2 18  software 
performance  growth  and  an  updated  mission  database  to  reflect  newly  developed  and/or 
refined  RA  designated  emitters,  RA  strategies,  and  jamming  techniques.  Re-test  Of  these 
improvements  were  planned  far  re-test  dating  a  return  visit  to  Western  Test  Range  in 
August  2003.  These  corrections  were  being  implemented  in  parallel  with  OA  and  the 
generation  of  the  OA  Report,  and  therefore  were  addressed  at  (he  NPDM  as  believed  to 
have  been  greatly  improved. 

•  "The  lack  of  a  fully  functional  keypad  prevented  the  test  aircrew  from  using  the  1CAP  01 
weapon  system  effectively.'’ 

o  A  Controls  and  Displays  Working  Group  (CDWG)  had  been  working  in  parallel  with 
OA  to  address  many  concerns  with  the  user  interface.  This  CDWG  concluded  in 
February  2003  with  recommended  software  changes  for  improved  keyboard  interface,  as 
well  as  modifications  to  die  keyboard  layout  to  place  the  roost  often  used  keys  in  the 
most  easily  accessible  place.  Tbe  NPDM  brief  clearly  indicated  significant  keypad 
improvements  were  made  in  May  2003.  Some  of  the  specific  changes  included 
grounding  improvements  between  the  keypad  and  displays  to  eliminate  display  lock-ups 
due  to  use  of  the  keypad,  corrections  of  PS/2  interface  codes  generated  by  tbe  keypad, 
display  software  improvements  to  eliminate  the  cursor  location  dependence  of  keypad 
operation,  improved  cursor  slew  speed  of  the  thumb  control,  and  user  interface 
improvements  to  link  up/down  arrows  and  PGUP/PGDN  keys  to  display  control.  These 
corrections  were  successfully  tested  in  DT  following  the  conclusion  of  OA. 

•  "Tbe  Tactical  EA-6B  Mission  Planning  System  erroneously  displayed  data  (lost  or  frozen 
data)  because  of  frequent  system  lockups,  which  presented  major  obstacles  to  effective 
preflight  mission  planning  and  post-fright  analysts." 

o  At  the  NPDM  brief,  the  Tactical  EA-6B  Mission  Planning  System  (TEAMS)  was 

presented  as  an  interim  solution  that  was  being  utilized  to  test  JCAP  01  in  absence  of  the 
Joint  Mission  Planning  System  (IMPS)  Framework  and  EA-6B  UPC.  Based  on  the  OA 
Report,  the  identified  path  forward  was  to  incorporate  only  the  most  critical  mission 
planning  improvements  to  support  OPEVAL,  as  the  JMPS  Framework  was  anticipated  to 
be  available  to  test  in  Follow-on  Test  and  Evaluation  (FOT&E)  prior  to  deployment. 

Note  tbe  mission  planner  is  a  separate  support  system,  and  not  integrated  as  part  0  tbe 
I  CAP  III  aircraft  system.  As  briefed  al  the  NPDM,  additional  planning  was  necessary  to 
further  define  mission  planner  functionality  to  be  included  in  OPEVAL  Additionally,  at 
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the  NPDM  brief.  Mission  Pi  Mining  was  specifically  called  ool  as  a  risk  item  being 
tracked 

•  “Fifty  additional  performance  deficiencies  fix  the  FCAP  III  systems  required  correction 
because  they  detracted  from  the  overall  EA-6B  efficiency  or  operator  situational  awareness  ” 

o  As  described  and  listed  in  die  NPDM  briefing,  alt  deficiencies  were  summarized  for 
ASNfRDA),  Of  the  50  non-major  risks  identified  by  COTF,  44  were  ”mu>or”  risk  items 
and  the  remaining  items  were  labeled  as  “other”.  The  briefing  to  ASN(RDA) 
acknowledged  these  additional  operational  concerns.  Given  the  majority  of  raised 
concents  were  to  be  addressed  via  software,  the  plan  presented  was  to  proceed  with  the 
hardware  procurement  and  address  all  significant  OA concerns  prior  to  entry  into 
OPEVAL,  But  as  described  in  the  three  prior  bullets,  significant  software  charges  were 
being  developed  in  parallel  to  OA  and  the  OA  Report  and  specifically  briefed  to 
ASNfRDA). 

Recommendation  A.U  We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  obtain  the  Commander,  Operational  Test  and  Evaluation  Force's 
operational  assessment  for  the  EA-6B  Improved  Capability  HI  and  not  approve  full -rate  production  until 
the  Commander,  Operational  Test  and  Evaluation  Force  determines  the  satisfactory  resolution  of  the  22 
critical  operational  issues. 

DO  NOT  CONCUR  WITH  THE  RECOMMENDATION 

1 ,  As  stated  above,  the  OA  Report  was  delivered  to  ASNfRDA)  as  part  of  the  LRIF  NPDM  process. 

2.  As  part  of  the  preparation  for  OPEVAL,  the  !CAP  ill  program  addressed  each  OA  risk  item  to 
ensure  that  improvements,  where  warranted,  were  included  in  the  product  provided  to  COTF  for 
OPEVAL  testing.  Since  a  determination  of  mission  effective  and  mission  suitable  (via  the 
OPEVAL  Report)  ts  identified  in  the  Acquisition  Decision  Memorandum  as  a  Full  Rale 
Production  (FRF)  criteria,  the  OA  Report  findings  should  be  superceded  by  the  OPEVAL  Report 
findings.  Therefore,  the  FRP  decision  should  be  based  on  resolution  of  any  OPEVAL  Report 
major  deficiencies,  not  areas  of  risk  identified  in  the  OA  Report. 

Recommendation  A.  2:  We  recommend  that  the  Program  Manager  for  the  EA-6B  analyze  and  identify 
frees  for  the  50  additional  deficiencies  identified  by  the  Commander,  Operational  Test  and  Evaluation 
Force  in  the  operational  assessment  to  increase  the  likelihood  that  the  fCAP  III  will  perform  satisfactorily 
before  concluding  the  dedicated  operational  test  and  evaluation  phase  of  the  acquisition  process, 

DO  NOT  CONCUR  WITH  THE  RECOMMENDATION 

It  is  the  responsibility  of  the  Program  Manager  to  address  both  major  and  minor  deficiencies  resulting 
from  operational  test,  but  not  in  the  manner  prescribed  by  this  recommendation.  The  fifty  areas  of  risk 
previously  cited  in  the  OA  report  (many  of  which  had  already  been  addressed)  and  those  deficiencies  that 
could  result  from  the  current  OPEVAL  are  managed  by  an  already  established  process  that  prioritizes 
them  (e.g.  Part  L  Part  2,  and  Part  3).  Based  on  those  priorities,  the  Program  Manager  in  consultation  with 
numerous  stakeholders  will  prepare  resolutions  to  any  identified  deficiencies.  The  resolution  can  range 
from  immediate  action  to  removing  the  requirement  in  a  future  requirement  update. 

The  Management  Control  process  used  by  PEO(T)  and  PMA-234  relies  on  a  comprehensive  reties  of 
reviews  and  evaluations  by  top-level  competency  managers  and  stakeholders  from  the  following 
Government  Codes: 
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•  PEO£T)OPS 

*  N78 

*  Training  PMA  205 

•  Safety,  AIR  4.1.10 

•  Environmental,  AIR  1.1E2 

*  Test,  Air  1.6 


•  N912 

•  VX-23 

•  Engineering  AIR  4.0B 

•  Software,  AIR  4.1.1 1 
4  Logistics,  AIR  3.0 

•  OSHA,  AIR  09F2 


•  DOT&E 

•  COTFOTC 

•  Systems  Engineering,  AIR  4.1 

•  Avionics,  AIR  4.5 

•  RAM,  AIR  41.6.1 


From  the  beginning,  all  deficiencies  were  presented  to  the  group  and  actions  identified  to  close  them  out. 
Actions  taken  ax  one  meeting  were  addressed  and  id  most  cases  dosed  or  dispositions!  by  the  next  review 
date,  The  Test  and  Engineering  communities  were  aware  of  the  status  of  all  deficiencies  and  monitored 
progress  through  the  review  cycle  leading  up  to  the  LRIP  decision.  A  list  of  the  reviews  used  by  PEOfT) 
and  PMA-234  to  assess  program  readiness  are  listed  below; 

•  1C  AP  111  EXCOM ,  14  March  2003 

♦  ICAP  W  Program  Summit,  17  April  2003 

•  Acquisition  Coordination  Team  Meeting  #1 ,  March,  2003 

*  Acquisition  Coordination  Team  Meeting  #2, 1 8  April,  2003 

♦  Acquisition  Coordination  Team  Meeting  #3, 23  April,  2003 

•  ICAP  TO  Follow-up  Meeting,  30  April,  2003 

*  ICAP  IB  Program  Status  Review,  7  May.  2003 

♦  ICAP  III  Pit’ Acquisition  Review  Board  (ARB).  May,  2003 

•  ICAP  UI  Acquisition  Review  Board  (ARB),  6  June,  2003 

♦  PieTCAPinNPDM.il  June,  2003 

•  EA-6B  ICAP  BINPDM.  12  June,  2003 

In  addition  to  addressing  deficiencies  during  the  LRIP  NPDM  and  OPEVAL  preparation  process  and  after 
further  investigations  of  the  risk  hems,  there  were  some  instances  where  actual  system  fixes  were  not 
required  In  these  cases  a  clarification  of  information  and/or  improved  training  was  provided  at  the 
Operational  Test  Readiness  Review  (OTRR).  Additionally.  COTF  identifies  and  prioritizes  deficiencies 
but  funding  to  fix  them  is  allocated  by  N78.  the  resource  sponsor. 

One  example  of  an  area  not  specifically  addressed  by  a  correction  is  the  decision  to  not  modify  the  ami- 
collision  light  design  because  further  research  indicated  it  to  be  an  isolated  incident  that  was  deemed 
addressable  by  training.  A  second  example  worth  noting  is  the  program  decision,  given  a  ccwt/bencfit 
tradeoff  foe  an  aircraft  nearing  end  of  life,  to  not  modify  a  cockpit  control  panel  fail  light  label  to  reflect 
“DSMLT  instead  of  the  legacy  “RRS";  again  this  would  be  addressed  via  training. 
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Finding  B  An  Interim  Authority  to  Operate  Improved  Capability  III  InfornuUon  Systems 

In  May  2003.  the  Designated  Approving  Authority  for  the  Naval  Air  System*  Command  issued  the 
Program  Manager  an  Interim  Authority  to  Operate  (1ATO)  the  1CAP  01  information  system  without 
requiring  the  Program  Manager  for  the  EA-6B  aircraft  to  first  complete  the  verification  and  validation 
phases  of  the  Deportment  of  Defense  Information  Technology  Security  Certification  and  Accreditation 
Process  (DITSCAP),  as  required.  As  a  result,  the  Program  Manager  began  operational  testing  of  a  system 
that  may  not  satisfy  system  information  assurance  requirements. 

PARTIALLY  CONCUR  (IN  PRINCIPLE)  WITH  THIS  FINDING 

The  Naval  Air  Systems  Command  (NAVAIRSYSCOM)  Designated  Approving  Authority  (DAA)  and 
Certification  Authority  have  reviewed  the  IATO  and  the  process  followed  in  issuing  this  document  by  the 
Information  Assurance  Program  Office  (IAPO)  and  note  the  following:  Although  we  concur  in  principle 
that  DITSCAP  typically  requires  the  review  and  issuance  of  an  IATO  after  completion  of  the  verification 
and  validation  phase  of  a  program,  we  note  that  specific  to  the  requirements  of  this  program,  the 
originator’s  interpretation  of  the  DITSCAP  does  not  accurately  reflect  the  practical  application  of  an 
Information  Assurance  (1A)  Program. 

NAVAIRSYSCOM  IA  Program  Office  met  with  the  ICAP  01  Program  to  address  overall  lA 
requirements,  to  include  testing  the  ICAP  01  variant  on  a  test  platform.  Note  this  platform  is  not 
“operational"  but  is  a  test  platform  in  a  controlled  test  environment.  The  lA  Program  Office,  an  office 
independent  of  PMA-234,  assessed  the  security  architecture  and  pasture  of  the  design  and  determined 
there  was  minimal  risk  of  unauthorized  disclosure,  modification,  or  availability  of  National  Security 
Information.  The  USN  and  DoD  have  traditionally  used  an  IATO  to  grant  temporary  authority  to 
programs  to  “operate”  systems  for  the  purpose  of  tearing  in  completion  of  the  verification  phase  of  the 
DITSCAP.  The  purpose  of  the  testing  is  to  identify  any  deficiencies  either  in  functionality  or  security. 
The  purpose  of  an  IATO  u  to  assess  the  risks  of  operating  the  “system"  in  order  to  complete  the 
verification  and  validation  efforts.  If  an  unacceptable  level  of  risk  to  the  protection  of  National  Security 
Information  bad  been  determined  during  this  review  process,  the  NAVAIRSYSCOM  Developmental 
DAA  would  not  have  authorized  the  IATO  for  the  test  and  evaluation  period. 

^Recommendation  B,  L:  We  recommend  that  the  Certification  Authority  for  the  EA-6B  Improved 
Capability  01  Program,  Naval  Air  Systems  Command: 

a  Complete  all  definition,  verification,  and  validation  requirements  tocomply  with  the  DoD 
Information  Technology  Security  Certification  and  Accreditation  Process. 

b.  Document  the  certification  procedures  and  tests  results  m  the  EA-6B  Improved  Capability  01 
System  Security  Authorization  Agreement. 

c.  Use  the  results  from  completing  the  DrTSCAP  to  recommend  to  the  Designated  Approving 
Authority  for  dre  EA-6B  Improved  Capability  01  whether  to  continue  granting  the  EA-6B 
Program  Manager  an  Interim  Authority  to  Operate. 

CONCUR  WITH  THIS  RECOMMENDATION 

Note  that  this  was  and  is  the  plan  of  the  NAVAIRSYSCOM  IA  Program  Office  and  Program  Manager 
and  was  briefed  as  such  during  the  audit. 
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Recommendation  B.  2.:  We  recommend  that  the  Designated  Approving  Authority  for  the  EA-6B 
Improved  Capability  Ill  Program,  Naval  Air  Systems  Command  base  the  decision  to  continue  granting  the 
EA-6B  Program  Manager  an  Interim  Authority  to  Operate  on  the  recommendation  of  the  Certification 
Authority  for  the  EA-6B  Improved  Capability  HI  Program. 

CONCUR  WITH  THIS  RECOMMENDATION 

Finding  C  Use  of  Failure  Dote  for  the  Improved  Capability  III  Program  Hardware 

The  subcontractor  did  not  submit,  through  the  prime  contractor  for  the  ICAP  ID  Tactical  Jamming  System 
Receiver,  updated  reliability  prediction  data  needed  to  perform  a  cost-benefit  analysis  to  determine  the 
best  maintenance  and  logistical  support  strategy  for  the  receiver.  This  condition  occurred  because  the 
Program  Manager  did  not  direct  the  prime  contractor  to  update  reliability  predictions  after  analyzing  and 
resolving  all  hardware  failures  identified  during  developmental  testing  or  to  retain  documentation  of 
corrective  actions  taken  to  reduce  the  frequency  of  hardware  failures.  As  a  result,  the  Program  Manager 
is  not  able  to  accurately  predict  the  expected  reliability  of  the  tactical  Jamming  System  Receiver  and  may 
incur  higher  than  expected  costs  to  maintain  and  logisticolly  support  the  receiver  when  the  ICAP  HI 
Program  becomes  operational. 

DO  NOT  CONCUR  WITH  THE  FINDING 

The  contractor's  Reliability  and  Maintainability  (RAM)  program  followed  during  the  EMD  period 
formed  the  basis  for  follow-on  phases  of  the  program  and  was  established  with  the  expressed  purpose  of 
identifying  failures  during  the  earliest  phase  io  the  design  and  to  implement  the  corrective  action  at  the 
earliest  possible  time  and  at  the  kast  cost  to  tbc  program.  Details  of  the  RAM  program  are  in  the 
contractor's  Integrated  Support  Plan .  The  program  includes  an  RAM  Program  Plan,  allocations, 
predictions,  development  testing,  FMECA.  Damage  Modes  Analysis,  Maintainability  Demonstration, 
Failure  Reporting  and  Corrective  Action  System  (FRACAS)  and  integration  into  the  LSAR,  The  Program 
Manager  actively  pursued  a  policy  ihat  ensured  that  the  supportability  and  testability  of  all  ICAP  III 
components  would  be  traceable  to  Failure  Modes  and/or  Damage  Modes  as  predicted  and  that  every 
corrective  action  would  be  accurately  documented  with  feedback  provided  back  to  the  design  agents. 

The  data  requested  by  the  audit  team  covered  a  time  period  of  over  two  years  and  would  have  needed  to 
be  formatted  info  a  usable  report  to  be  useful  to  the  team.  By  mutual  consent  the  audit  team  and  the  ICAP 
III  IPT  decided  against  producing  such  a  report  because  it  would  constitute  creating  a  new  document  and 
therefore  would  be  outside  the  scope  of  existing  program  documentation  The  ICAP  ID  Reliability 
Engineer  met  with  the  audit  team  on  May  3,  2004  to  discuss  the  methodology  used  to  perform  an  RAM 
analysis.  During  a  follow-up  meeting  with  the  audit  team,  the  RAM  Engineer  provided: 

1).  Northrop  Grumman  FRACAS  database  examples.  Examples  were  from  development 
testing  at  each  contractor  facility. 

2}.  Reliability  Maintainability  Review  Board  (RMRB)  database.  This  database  correlates  to 
the  individual  FRACAS  databases  that  reside  at  each  contractor  facility.  The  purpose  of 
this  database  is  to  provide  a  repository'  for  all  failures  experienced  in  the  development  life 
of  the  program.  It  provides  Relevancy  /  Non-relevancy  definitions,  failure  analysis  and 
corrective  actions. 

3),  RMRB  meeting  minutes  and  documentation.  RMRB  meetings  ore  held  quarterly  at  each 

contractor  facility.  The  purpose  of  the  RMRB  process  is  to  disposition  failures  (Relevant  / 
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Non-Retevam)  in  accordance  predetermined  ground  rules  (c  bar  are  in  the  minutes)  and 
provide  failure  analysis  and  corrective  actions  for  all  relevant  failures. 

Recommendation  C:  We  recommend  that  the  Program  Manager  for  the  EA-6B  Program: 

1  Require  Northrop  Grumman  Electronic  Systems,  Baltimore,  Maryland;  through  the  prime 
contractor,  to  update  the  failure  report  logbook  to  prioritise  open  system  failures  for  the  Tactical 
Jamming  System  Receiver  and  to  cross-reference  corrective  actions  taken  10  close  each  failure 
entry, 

DO  NOT  CONCUR  WITH  THE  RECOMMENDATION 

The  FRACAS  provides  the  means  to  document  and  improve  the  reliability  of  the  design  through 
failure  recording,  analysis  and,  if  applicable,  corrective  design  actions.  FRACAS  reporting  does 
not  begin  until  the  system  reaches  design  maturity.  In  the  case  of  the  lCAPUJ  program,  the  RAM 
IPT  determined  that  FRACAS  reporting  would  begin  when  the  system  was  sufficiently  mature. 
This  approach  is  per  Attachment  5,  paragraph  5  .2.8  of  the  1CAP  ID  contract  to  implementation  of 
a  FRACAS  program.  Logbook  entry  anomalies  may  or  may  not  be  in  the  FRACAS  database 
depending  on  the  phase  in  development  in  which  the  anomalies  occurred.  Logbooks  are  an 
internal  Northrop  Grumman  practice  and  not  a  contract  requirement.  The  purpose  of  the 
logbook!  s)  is  to  record  events  that  occur  during  each  shift  If  these  entries  where  relevant  to  a 
failure  of  the  system  coder  test  and  occurred  after  design  maturity  had  been  achieved  by  mutual 
IPT  agreement,  those  entries  would  be  found  in  the  FRACAS  database.  Otherwise  (bey  are 
intentionally  outride  of  the  FRACAS  process  and  mote  accurately  are  described  as  development 
notes. 

Z.  Obtain  from  Northrop  Grumman  Electronic  Systems,  Baltimore,  Maryland,  through  the  prime 
contractor,  an  updated  failure  modes  and  effects  and  criticality  analysis  document  that  includes  an 
analysis  of  corrective  actions  taken  to  resolve  system  failures  that  occurred  after  June  2001 . 

DO  NOT  CONCUR  WITH  THE  RECOMMENDATION 

Both  the  Reliability  Prediction  and  the  FMECA  are  design  tools  used  to  assess  the  proposed 
design  for  compliance  with  design/program  requirements.  They  are  also  used  for  the  initial 
assessment  of  supporlability  and  testability.  The  predictions  are  design  assessment  tools  used  in 
the  absence  of  actual  failure  data.  Once  they  are  complete  and  the  design  Is  solidified  there  is  Little 
payoff  in  updating  them  unless  major  design  changes  occur.  They  should  not  be  updated  to  reflect 
actual  failure  data  since  actuals  are  better  than  predictions  and,  once  actuals  are  obtained,  the 
predictions  are  obsolete.  There  would  be  no  value  added  in  requiring  the  contractor  to  update  the 
FMECA  at  this  stage  of  the  Program,  given  the  small  sample  rate  experienced  at  that  lime. 

3.  Perform  a  cost-beoefil  analysis  after  implementing  Recommendations  C.l  and  C.2  k>  determine 
the  roost  cost-effective  acquisition  support  strategy  for  the  Tactical  Jamming  System  Receiver. 

DO  NOT  CONCUR  WITH  THE  RECOMMENDATION 

The  Business  Case  Analysis  (BCA)  is  performed  by  NAVICP,  which  is  tm  independent  program 
office  not  under  the  control  of  the  ICAP ITI  IPT  or  the  EA-6B  Program  Manager.  The  1CAP  111 
ILS  IPT  has  provided  NAVICP  with  the  latest  Reliability  predictions  for  use  in  the  BCA  and  is  in 
(be  process  of  revising  those  predictions  based  on  experience  to  dale  in  TECHEVAL  and 
OPEVAL.  Tbe  operating  hours  sample  size  is  ftc*  yet  statistically  significant  enough  to  be  used  in 
lieu  of  the  predictions.  The  predictions  can  only  be  tempered  using  engineering  judgment  based 
on  the  few  failures  Incurred  to  date.  The  predicted  reltabi  lity  being  used  for  forecasting  support 
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require  meats  spares  requirements.  Level  Of  Repan  Analysis  (LORA),  and  numerous  cttier  support 
related  analysis  -  including  the  BCA  -  are  defraud  8  to  1  (multiplied  by  12.5%)  from  the 
predicted  values.  This  derate  factor  is  based  in  NAVAIR  RAM  competency  actual  experience 
with  similar  systems  over  its  many  years  of  evaluating  design  of  Navy  Weapon  systems. 

DON  Commcm  to  Append!.  A:  Scope  end  Melhadolop.  Manatownl  Control  Progom  Btrkw: 

Naval  Inspector  General  memorandum,  dated  18  June,  20CW  accompanied  the  delivery  of  the  Report  on 
Acquisition  of  the  EA-dB  Improved  Capability  II)  Program  (Project  No.  02003 AE -0190).  In  paragraph 
four,  the  memo  states,  “Management  should  also  comment  on  the  material  management  control 
weaknesses  discussed  in  Appendix  A*\  In  a  separate  correspondence,  the  NAVAIR  K3  Audit  Liaison 
Officer  directed  PMA-234  to  respond  to  the  management  control  issues  raised  in  Appendix  A. 

The  following  is  PMA  234’s  response  to  the  Management  Control  Program  Review  section  in  Appendix 
A  in  particular  the  subsection  tilled,  "Adequacy  of  Management  Controls  “ 


Adequacy  of  Management  Controls  (Appendix  A,  page  21) 

We  identified  management  control  w  eaknesses,  as  defined  by  DoD  Instruction  5010.40,  relating  to 
reporting  of  rest. and  evaluation  results,  information  assurance,  and  system  reliability  failure  reviews. 

decision  authority  as  required  for  tbe  biyy  program  tteiskm  idcpMH  Of  fvHv  VfcBfY.lM*  ffwtojg  md 
corrective  action  had  been  performed  on  hardware  fa  hires,  and  Ite-Ifei  amed  Approving  Atfftontt.fer 
the  Naval  Air  System  Command  did  not  follow  procedure  for  ismi? m  Utt  Ml  1AXQ- 

Recommendations  A.l,  A.2,  B.  1.  B.2.  C.l,  aod  C2t  if  implemented,  will  improve  the  overall  management 
of  the  ICAP  IB  Program.  A  copy  of  this  report  will  be  provided  to  the  senior  official  responsible  for 
management  controls  in  the  Department  of  the  Navy. 


DO  NOT  CONCUR  (underlined  portion) 

-I.  The  Program  Manager  did  not  provide  the  COTF  operational  assessment  to  the  milestone  decision 
authority  as  required  for  the  navy  program  decision  meeting. 

Response:  ASN(RDA)  records  show  receipt  of  the  COTF  assessment  report,  see  Enclosure  (3). 
Furthermore,  all  of  the  issues  tod  concerns  highlighted  in  the  assessment  were  covered  during  the 
reviews  leading  up  to  the  meeting  with  ASN(RDA).  Tbe  Program  Manager  closely  monitored 
TECHEVAL  progress  and  instructed  the  ICAP  HI  TAE  team  to  conduct  daily  phone  comms  to 
monitor  progress  and  corrective  actions.  TECHEVAL  issues  were  a  topic  of  discussion  during 
every  ICAP  HI  briefing  leading  up  to  and  including  the  NPDM.  COTF  personnel  were  in 
attendance  at  every  review  to  discuss  ibcrr  progress,  deficiencies,  and  actions  taken  to  date. 

Action  items  generated  at  one  review  were  expected  to  be  corrected  or  disposilioned  by  tbe  next 
review  date  where  they  would  be  reviewed  by  tbe  attendees.  Any  weaknesses  identified  during 
the  reviews  were  identified  and  assigned  as  actions  for  the  follow-on  review.  Each  review  was 
attended  by  representatives  from  tbe  following  competencies  and  Government  agencies  to  ensure 
that  the  program  did  not  go  forward  unless  it  met  its  cost,  schedule,  and  performance!  objectives. 


•  PEOfDOPS 

•  N78 

•  Training  PMA  205 

•  Safety,  AIR  4.1.10 

•  Environmental,  AIR  1.IE2 


a  mi2 

•  VX-23 

•  Engineering  ADR  4.0B 

•  Software,  AIR  4.1.11 

•  Logistics,  AIR  3,0 


•  DOTAE 

•  COTF OTC 

•  Systems  Engineering,  AIR  4. 1 
«  Avionics,  AIR  4,5 

•  RAM,  AIR  4  1.6.1 
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•  Test*  Air  1.6  *  OSHA.A1R09F2 

PEO(T)  records  iodic  ale  the  following  Management  Control  CMC)  reviews  look  place  at  winch 
time  competency  managers,  FEO(T)  staff*  and  PMA-234  personnel  conducted  detailed  discussions 
on  the  information  that  would  eventually  be  documented  in  the  COMOPTEVFOR  Report.  The 
MC  reviews  include: 

•  1CAP  m  EX  COM*  14  March  2003 

•  ICAP  m  Program  Summit,  17  April  2003 

•  Acquisition  Coordination  Team  Meeting  #1,  March,  2003 

•  Acquisition  Coordination  Team  Meeting  #2*  18  April,  2003 

•  Acquisition  Coordination  Team  Meeting  #3, 23  April,  2003 

•  ICAP  m  Follow-up  Meeting.  30  April.  2003 

•  1CAP  HI  Program  Status  Review*  7  May,  2003 

•  ICAP  HI  Prc-Acquisitioo  Review  Board  (ARB),  May,  2003 

•  ICAP  111  Acquisition  Review  Board  (ARB),  6  June,  2003 

•  Pro- ICAP  D1 NPDM,  1  1  June,  2003 

•  EA-6B  ICAP  m  NPDM,  12  June,  2003 

2.  The  Program  Manager  did  not  folly  verify  that  analysis  and  correcti  ve  action  had  been  performed 
on  hardware  failures. 

Response:  The  Program  Manager  contracted  for  and  instituted  a  closed  loop  failurcAroublc 
corrective  action  process  that  ensured  that  all  data  collected  during  the  T&£  process  would  be 
considered  in  the  corrective  action  process  and  would  be  fed  into  any  redesign  effort. 

The  true  test  of  whether  the  Program  Manager’s  policy  was  effective  is  revealed  in  the  Hardware 
maturity  status  reported  out  during  the  MC  reviews  and  was  a  key  Assessable  Unit  (RDT&E 
Performance  Evaluation)  tracked  by  PEOfT).  The  results  of  that  assessment  indicated  a  hardware 
maturity  that  met  or  exceeded  the  requirements: 

With  190.7  aircraft  flight  hours  and  over  3000  operating  hours  on  equipment  since  the  start  of  the 
test  program  the  system  experienced  a  MTTR  of  46  minutes  with  a  requirement  of  <90  minutes 
and  a  MTBF  of  209.7  hrs  with  a  requirement  of  209  hr*.  All  deficiencies  and  problem  reposts 
were  disposiiiooed  as  the  time  the  system  entered  OPEVAJL  This  is  primarily  due  to  the 
comprehensive  corrective  action  plan  implemented  by  the  Program  Manager  at  the  start  of  the 
program. 

3,  The  Designated  Approval  Authority  (DAA)  for  the  Naval  Air  Systems  Command  did  not  follow 
procedures  for  issuing  the  Program  Manager  an  I ATD. 

Response:  The  DAA  followed  the  accurate  interpretation  of  the  DITSCAP  program.  A  thorough 
analysis  of  the  system  architecture  was  conducted  and  follow-up  discussions  with  the  ICAP  HI 
Technical  team  resulted  in  the  assessment  that  the  ICAP  ID  was  at  low  risk  of  breaching 
information  Assurance  (IA)  criteria  as  a  platform  IT  interconnection  system. 

Furthermore,  ICAP  HI  achieved  Phase  !  DITSCAP  and  received  an  IATQ  containing 
requirements  that  are  being  accomplished  at  this  time.  The  DAA  imposed  the  requirements  on 
ICAP  HI  in  order  to,  ''proceed  towards  full  accreditation  of  the  ICAP  HI  and  subordinate  efforts. 

The  ICAP  HI  Program  Team  has  ongoing  discussions  with  the  DAA  and  is  in  the  process  of 
completing  a  system  level  SSAA  for  his  evaluation. 
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Peter  C.  Johnson 
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